adds colors and images to readme; adds ANALYSIS.md

ANALYSIS.md

@@ -0,0 +1,166 @@
+# Prisma 357M — Spectral and Representation Analysis
+
+Post-training analysis of the Prisma 357M checkpoint. All plots generated from the final model weights and activations on a sample of WikiText-103 validation data.
+
+Scripts: `scripts/spectral_analysis.py`, `scripts/representation_analysis.py`
+
+---
+
+## Representation Analysis
+
+### CKA Self-Similarity
+
+Centered Kernel Alignment between all layer pairs. Measures whether two layers encode similar representational structure regardless of rotation or scaling.
+
+![cka_self](scripts/representation_output/prisma/cka_self.png)
+
+The matrix reveals three distinct processing regimes separated by sharp boundaries:
+
+- **Expand E0-E7** (top-left bright block): High mutual similarity — these layers incrementally refine the embedding representation. The model stays close to input space.
+- **Expand E8-E19** (second block): A different representational regime emerges around E8. These layers are similar to each other but dissimilar to both early expand and compress layers — the model has moved into an abstract internal space.
+- **Middle M0**: Near-zero CKA with almost everything. The single middle layer is a representational bottleneck — it transforms between expand and compress coordinate systems.
+- **Compress C0-C19** (bottom-right block): The compress phase rebuilds similarity gradually. Early compress layers (C0-C5) are transitional; C6+ form their own coherent block.
+
+The off-diagonal structure is telling: expand layers have essentially zero CKA with late compress layers, confirming that the two phases operate in genuinely different representational spaces despite sharing weights. The gates (W3/W4) are doing real work — same W1/W2 matrices produce completely different representations depending on direction.
+
+
+### Logit Lens
+
+Projects intermediate representations through the output head at each layer to see when the model "knows" the answer.
+
+![logit_lens_summary](scripts/representation_output/prisma/logit_lens_summary.png)
+
+Four views of prediction formation across the 41-layer pipeline:
+
+- **Prediction entropy** (top-left): Near-zero through the expand phase — representations are close to embedding space, so the output head produces confident (but wrong) predictions based on surface similarity. Entropy spikes at the middle layer and stays high through compress. The model is doing real work — disrupting surface-level confidence to build correct predictions.
+- **Top-1 probability** (top-right): Mirror of entropy. High confidence in expand (misleading), drops at middle, partially recovers in compress as correct predictions crystallize.
+- **Median rank of correct token** (bottom-left): The correct token starts at rank ~10,000 (out of 32K vocabulary) at the embedding layer and drops to rank ~1 by the final compress layers. The expand phase brings it from 10K to ~30 (coarse semantic neighborhood); the compress phase refines from ~30 to 1 (precise token selection).
+- **Convergence toward final prediction** (bottom-right): Agreement with the model's actual output. Near-zero throughout expand — the model hasn't committed to an answer. Begins climbing at C0 and reaches 1.0 by C19. The compress phase IS the decision-making process.
+
+The logit lens confirms the architectural hypothesis: expand builds abstract representations (moving away from token space), compress converts them back into token predictions (moving toward output space).
+
+
+### Representation Drift
+
+How much each layer changes the representation relative to its predecessor.
+
+![representation_drift](scripts/representation_output/prisma/representation_drift.png)
+
+- **Cosine similarity** (left): E0 makes the biggest directional change (0.68) — the first layer reorients the embedding substantially. Subsequent expand layers are increasingly gentle (0.93-0.99). The middle layer drops to 0.89 — another major reorientation. Compress layers stay high (0.93-0.98), making incremental adjustments.
+- **L2 distance** (right): E0 has the largest magnitude change (~29). Expand layers settle to small updates (5-15). Compress layers show gradually increasing L2 distance (15-35) — each compress layer makes a larger magnitude adjustment than the last, consistent with the progressive refinement visible in the logit lens. The final norm (gray, ~85) applies the largest single transformation, collapsing the representation to output scale.
+
+The asymmetry between cosine (high = small angular change) and L2 (growing through compress) suggests that compress layers maintain the representational direction established by expand while progressively scaling and sharpening specific features for token prediction.
+
+---
+
+## Spectral Analysis
+
+### Activation Effective Rank Progression
+
+Effective rank of activation covariance matrices at each layer — measures the dimensionality of the representation (how many independent directions carry meaningful variance).
+
+![activation_rank_progression](scripts/spectral_output/prisma/activation_rank_progression.png)
+
+The hourglass is visible in the numbers:
+
+- **Embedding**: erank ~215 (the input dimensionality baseline)
+- **Expand E0-E6**: erank ~220, slightly above embedding — the representation briefly expands
+- **Expand E7-E16**: Gradual decline from ~220 to ~170 — progressive abstraction compresses the representation
+- **Expand E17-E19**: Sharp collapse to ~45 — the final expand layers aggressively compress toward the bottleneck
+- **Middle M0**: erank ~50 — the bottleneck. The entire model's information passes through ~50 effective dimensions
+- **Compress C0-C2**: Rapid recovery to ~75, ~130, ~190
+- **Compress C3-C19**: Climbs to ~230, exceeding the embedding baseline — the compress phase reconstructs a richer representation than the input
+
+The compress phase doesn't mirror the expand phase — it *overshoots*, producing higher effective rank. This makes sense: the output representation needs to distinguish between 32K tokens, requiring more dimensions than the embedding's initial encoding.
+
+
+### Activation Eigenspectra
+
+Eigenvalue distributions and cumulative variance concentration across all layers.
+
+![activation_spectra](scripts/spectral_output/prisma/activation_spectra.png)
+
+- **Left (eigenvalue distribution)**: Expand layers (blue) show flatter spectra (more distributed variance), middle layers have steep spectra (concentrated variance), compress layers (red) rebuild distributed spectra. The spectral shape changes continuously through the pipeline.
+- **Right (cumulative variance)**: The middle layer concentrates 90% of variance in ~50 components. Expand and compress layers need 150-200+ components to reach 90%. This confirms the bottleneck isn't just in effective rank — the actual information content is compressed.
+
+
+### Mirror Pair Activation Comparison
+
+Activation spectra for each mirror pair, comparing expand vs compress phases. Each subplot shows one shared-weight pair operating in both directions.
+
+![mirror_pair_comparison](scripts/spectral_output/prisma/mirror_pair_comparison.png)
+
+Despite sharing W1 and W2, expand and compress activations have different spectral profiles. The gates (W3/W4) reshape the spectral distribution without changing the structural transformation. Earlier pairs show larger expand/compress divergence; later pairs converge — consistent with the CKA finding that late expand and early compress layers are the most dissimilar.
+
+
+### Embedding vs Final Activation Spectra
+
+Direct comparison of the embedding matrix spectrum and the final-layer activation spectrum.
+
+![embedding_alignment](scripts/spectral_output/prisma/embedding_alignment.png)
+
+- **Embedding**: erank 955, nearly flat spectrum — the frozen MobileLLM embeddings distribute information broadly across all 1024 dimensions with minimal concentration.
+- **Final activation**: erank 218, steep spectrum — the model has learned to concentrate its output into ~218 active dimensions, with the top 25 components carrying ~50% of variance.
+
+The model takes a broadly distributed input signal and progressively concentrates it into a lower-dimensional but more structured output. The 4.4x rank reduction (955 to 218) is the spectral signature of the expand-compress pipeline.
+
+
+### G2LU Gate Spectra (W3 vs W4)
+
+Weight spectra and effective rank of the outer gate (W3) vs inner gate (W4) across all 21 mirror pairs.
+
+![gate_spectra](scripts/spectral_output/prisma/gate_spectra.png)
+
+- **Top (weight spectra)**: W3 and W4 have nearly identical spectral shapes — both use the full rank of the weight matrix. Neither gate has collapsed or become low-rank. The nested gating structure (W4 modulates W3) doesn't force one gate to become simpler than the other.
+- **Bottom (effective rank per pair)**: Both gates maintain erank ~900 (out of 1024) across all pairs, with W4 (inner) slightly lower than W3 (outer) in most pairs. The ~50-rank gap is consistent: the inner gate uses slightly fewer effective dimensions, potentially because it operates as a coarser filter that the outer gate refines.
+
+
+### Layer-wise Spectral Properties (FFN W1)
+
+Spectral metrics for the shared W1 projection matrix across all mirror pairs.
+
+![layer_progression](scripts/spectral_output/prisma/layer_progression.png)
+
+- **Effective rank** (top-left): Uniformly high (~930) across all layers — no weight collapse anywhere.
+- **Stable rank** (top-right): Peaks sharply at pairs 10-11 (the layers nearest the architectural midpoint) with stable rank ~50, compared to ~20-30 for other pairs. The midpoint layers have the most distributed singular value spectra — the most "general" transformations.
+- **Power-law alpha** (bottom-left): All layers below the alpha=2 boundary (heavy-tailed), indicating structured, non-random weight matrices. Remarkably uniform across layers (~0.30-0.35).
+- **Signal ratio** (bottom-right): Very low (<0.006) — almost all singular values fall below the Marchenko-Pastur noise bound. This is consistent with the distributed, non-sparse nature of the weights; information is encoded in the collective spectrum rather than in isolated large singular values.
+
+
+### Spectral Comparison: Prisma vs GPT-2 Medium
+
+Layer-by-layer comparison of spectral properties between Prisma 357M and GPT-2 Medium (355M).
+
+![comparison](scripts/spectral_output/prisma/comparison.png)
+
+- **Effective rank**: Nearly identical (~930-950) for both models across all layers. The mirrored architecture doesn't sacrifice weight expressiveness.
+- **Stable rank**: Prisma shows more variation between layers (range ~20-50) vs GPT-2's flatter profile (~25-35). The midpoint peak in Prisma has no equivalent in GPT-2 — it's a structural consequence of the mirrored architecture.
+- **Power-law alpha**: Both models in the same range (0.25-0.35), both heavy-tailed. Prisma's alpha is slightly more uniform, potentially reflecting the regularizing effect of weight sharing.
+
+
+### Weight Spectra by Component
+
+Individual weight spectra (linear and log-log scale) for each parameter type across all layers.
+
+#### Embedding
+![weight_spectra_embedding](scripts/spectral_output/prisma/weight_spectra_embedding.png)
+
+The frozen MobileLLM embedding has one dominant singular value (~115) with a steep drop to ~18 for the second, then a gradual tail. The Marchenko-Pastur bound (dotted line at ~35) shows only 1 singular value above the noise floor — the embedding matrix is effectively rank-1 plus structured noise. This is the "fixed coordinate system" that anchors the entire model.
+
+#### FFN Shared Projection (W1)
+![weight_spectra_ffn_W1](scripts/spectral_output/prisma/weight_spectra_ffn_W1.png)
+
+All 21 W1 matrices have similar spectral profiles: top singular value ~16, smooth decay. Tight clustering in log-log confirms that shared weights don't develop pathological layer-specific structure.
+
+#### Outer Gate (W3)
+![weight_spectra_ffn_gate_W3](scripts/spectral_output/prisma/weight_spectra_ffn_gate_W3.png)
+
+#### Inner Gate (W4)
+![weight_spectra_ffn_gate_W4](scripts/spectral_output/prisma/weight_spectra_ffn_gate_W4.png)
+
+W3 and W4 are spectrally near-identical at the weight level — confirming that their functional differentiation (inner vs outer gate) emerges from the *compositional structure* of G2LU rather than from divergent weight distributions. The nested relationship `silu(W3@x * silu(W4@x))` creates functional asymmetry from structural symmetry.
+
+#### Attention Q Projections
+![weight_spectra_attention_Q](scripts/spectral_output/prisma/weight_spectra_attention_Q.png)
+
+Q projections show slightly more inter-layer variance than FFN weights — different layers attend to different things, while FFN transformations stay more uniform. The log-log tail follows a clean power law across all layers.

README.md

@@ -7,27 +7,20 @@ language:
 - en
 pipeline_tag: text-generation
 ---
+<div align="center">
+
 # Prisma
+![prisma](img/prisma.png)
+
+</div>
+
+---
 
 A prototype model that is assembled as a mirrored transformer architecture with nested gating (adds an extra weight to the FFN) and morphological position encoding. It proposes that the model architecture creates different scaffolding, leading to different training regimens and capabilities.
 
 Prisma is only viable as it piggybacks on pre-trained tokenizers and their weight-tied embeddings, it decomposes the transformer architecture into symmetric **expand** and **compress** phases that share structural weights, connected by a small number of unique **middle** layers. Information expands from tokens to semantics, then compresses back — like light through a prism.
 
-```
-Token Embeddings
-      |
-  [ Expand  ]  ─── mirror pair 1 (W1, W2 shared) ── G²LU gate (W3·W4)
-  [ Expand  ]  ─── mirror pair 2
-  [  ....   ]  ─── mirror pair N
-      |
-  [ Middle  ]  ─── unique layers (full capacity, not shared)
-      |
-  [Compress ]  ─── mirror pair N (same W1, W2 as expand N)
-  [Compress ]  ─── mirror pair 2
-  [Compress ]  ─── mirror pair 1
-      |
-   LM Head (weight-tied to embeddings)
-```
+![architecture](img/prisma_arch.png)
 
 
 ## Key Concepts
@@ -66,7 +59,7 @@ The regularization stack (mirroring + G²LU + WoRPE) enables training at **100x
 
 ### ~350M scale prototype — comparison with published models
 
-Prisma 357M trained on ~30B tokens (OpenWebText 20% + FineWeb-Edu 10BT continued training), compared against published models at similar scale:
+Prisma 357M trained on ~30B tokens (OpenWebText 20% + FineWeb-Edu 10BT continued training), compared against published models at similar scale.
 
 | Model | Params | Train Data | ARC-C\* | ARC-E\* | BoolQ | HellaSwag\* | LAMBADA | PIQA\* | WikiText\*\* | WinoGrande |
 |---|---|---|---|---|---|---|---|---|---|---|
@@ -79,6 +72,11 @@ Prisma 357M trained on ~30B tokens (OpenWebText 20% + FineWeb-Edu 10BT continued
 
 \* *normalized accuracy* · \*\* *word perplexity*
 
+![training_budget](img/training_budget_vs_model.png)
+
+
+![bench_vs_model](img/bench_vs_model.png)
+
 **Key findings:**
 - **Beats GPT-2 medium on 5/8 benchmarks** (ARC-C, ARC-E, BoolQ, HellaSwag, PIQA) with 25% less training data.
 - **Beats Baguettotron (200B) on 6/8 benchmarks** — including PPL — with **7x less data.**
@@ -97,6 +95,13 @@ Prisma 357M trained on ~30B tokens (OpenWebText 20% + FineWeb-Edu 10BT continued
 | + continued (FineWeb c1) | 1e-3 | 0.249 | 0.434 | 0.601 | 0.333 | 0.312 | 0.626 | 34.7 |
 | + continued (FineWeb c2) | 1e-3 | 0.290 | 0.548 | 0.620 | 0.427 | 0.362 | 0.670 | 27.4 |
 
+#### Prisma 357M detailed training charts
+
+![training_phase_1](img/training_phase_1_openwebtext_20p_x4.png)
+
+![training_phase_2](img/training_phase_2_fineweb_edu_10B_x2.png)
+
+![bench_results_per_checkpoint](img/bench_results_vs_checkpoint.png)
 
 ## Quick Start
 
@@ -241,6 +246,8 @@ lm_eval --task arc_challenge,arc_easy,boolq,hellaswag,lambada_openai,piqa,wikite
 
 ## Architecture Details
 
+Full spectral and representation analysis with all plots: **[ANALYSIS.md](ANALYSIS.md)**
+
 ### Why Mirroring Works
 
 Mirroring only works due to the additional gate. W3 and W4 specialize to serve different roles despite sharing weights — spectral analysis confirms the gates swap their stable-rank profiles at the architectural midpoint. The order of mirror layers may be rearrangeable, as the gates adapt to whatever representations flow through them.

scripts/representation_output/prisma/results.json

@@ -0,0 +1,2587 @@
+{
+  "cka_self": {
+    "names": [
+      "embedding",
+      "expand_0",
+      "expand_1",
+      "expand_2",
+      "expand_3",
+      "expand_4",
+      "expand_5",
+      "expand_6",
+      "expand_7",
+      "expand_8",
+      "expand_9",
+      "expand_10",
+      "expand_11",
+      "expand_12",
+      "expand_13",
+      "expand_14",
+      "expand_15",
+      "expand_16",
+      "expand_17",
+      "expand_18",
+      "expand_19",
+      "middle_0",
+      "compress_0",
+      "compress_1",
+      "compress_2",
+      "compress_3",
+      "compress_4",
+      "compress_5",
+      "compress_6",
+      "compress_7",
+      "compress_8",
+      "compress_9",
+      "compress_10",
+      "compress_11",
+      "compress_12",
+      "compress_13",
+      "compress_14",
+      "compress_15",
+      "compress_16",
+      "compress_17",
+      "compress_18",
+      "compress_19",
+      "final_norm"
+    ],
+    "matrix": [
+      [
+        1.0,
+        0.8866188526153564,
+        0.8164767026901245,
+        0.7336245179176331,
+        0.7054738998413086,
+        0.6806746125221252,
+        0.6546079516410828,
+        0.6396195292472839,
+        0.5966438055038452,
+        0.085598886013031,
+        0.07780761271715164,
+        0.07064133882522583,
+        0.06587867438793182,
+        0.06444326043128967,
+        0.06620140373706818,
+        0.06643415987491608,
+        0.06744755059480667,
+        0.06798108667135239,
+        0.023102257400751114,
+        0.02318720705807209,
+        0.023029332980513573,
+        0.02303888462483883,
+        0.0230090469121933,
+        0.023262370377779007,
+        0.07920751720666885,
+        0.08516508340835571,
+        0.12063281238079071,
+        0.13757441937923431,
+        0.1504034847021103,
+        0.20180721580982208,
+        0.20416469871997833,
+        0.20618851482868195,
+        0.2103184014558792,
+        0.276023268699646,
+        0.2877357304096222,
+        0.27862244844436646,
+        0.2714858949184418,
+        0.2638208270072937,
+        0.24908611178398132,
+        0.23528440296649933,
+        0.2028241604566574,
+        0.18157896399497986,
+        0.21474751830101013
+      ],
+      [
+        0.8866188526153564,
+        1.0,
+        0.9628973603248596,
+        0.8799647688865662,
+        0.8539307713508606,
+        0.8308154940605164,
+        0.8077625632286072,
+        0.7751748561859131,
+        0.7229080200195312,
+        0.10106048732995987,
+        0.09334167093038559,
+        0.08555062115192413,
+        0.08125422894954681,
+        0.0780123695731163,
+        0.07894133031368256,
+        0.08002081513404846,
+        0.07889489084482193,
+        0.09398424625396729,
+        0.11173419654369354,
+        0.11162395775318146,
+        0.11166036128997803,
+        0.11173491179943085,
+        0.1118139699101448,
+        0.11213553696870804,
+        0.15420915186405182,
+        0.15838824212551117,
+        0.18471214175224304,
+        0.1943701058626175,
+        0.20828627049922943,
+        0.22415931522846222,
+        0.22729963064193726,
+        0.23028919100761414,
+        0.23264256119728088,
+        0.3063642978668213,
+        0.3168011009693146,
+        0.3035929203033447,
+        0.2928607165813446,
+        0.28051790595054626,
+        0.2587052285671234,
+        0.23799873888492584,
+        0.20439808070659637,
+        0.1831863820552826,
+        0.22291426360607147
+      ],
+      [
+        0.8164767026901245,
+        0.9628973603248596,
+        1.0,
+        0.9413407444953918,
+        0.9221710562705994,
+        0.9034830331802368,
+        0.8851162791252136,
+        0.8494495749473572,
+        0.7978618741035461,
+        0.11317962408065796,
+        0.10702235996723175,
+        0.09908141940832138,
+        0.09638337790966034,
+        0.09181856364011765,
+        0.09098811447620392,
+        0.0925474762916565,
+        0.08911333978176117,
+        0.1197425052523613,
+        0.2184595763683319,
+        0.2182939201593399,
+        0.21833431720733643,
+        0.21851415932178497,
+        0.21860040724277496,
+        0.21891941130161285,
+        0.23645025491714478,
+        0.23678229749202728,
+        0.2491903305053711,
+        0.2493284046649933,
+        0.2616477906703949,
+        0.23879699409008026,
+        0.24320976436138153,
+        0.2469138205051422,
+        0.24982884526252747,
+        0.3279764652252197,
+        0.33550044894218445,
+        0.3209203779697418,
+        0.30974993109703064,
+        0.2971762716770172,
+        0.2757277190685272,
+        0.2561028003692627,
+        0.22381751239299774,
+        0.20405621826648712,
+        0.24675846099853516
+      ],
+      [
+        0.7336245179176331,
+        0.8799647688865662,
+        0.9413407444953918,
+        1.0,
+        0.995227575302124,
+        0.9878040552139282,
+        0.9771004319190979,
+        0.9507975578308105,
+        0.9107967615127563,
+        0.308550626039505,
+        0.3040868937969208,
+        0.29570043087005615,
+        0.2935424745082855,
+        0.2864590287208557,
+        0.2842535078525543,
+        0.2867068648338318,
+        0.2797296643257141,
+        0.32049211859703064,
+        0.27435117959976196,
+        0.27240100502967834,
+        0.27254536747932434,
+        0.27274295687675476,
+        0.27290886640548706,
+        0.2748803198337555,
+        0.40867719054222107,
+        0.41179877519607544,
+        0.4371880888938904,
+        0.43606802821159363,
+        0.4444027245044708,
+        0.38802891969680786,
+        0.3861248791217804,
+        0.3806435763835907,
+        0.3713274300098419,
+        0.3279266059398651,
+        0.3327653408050537,
+        0.3181270658969879,
+        0.30604374408721924,
+        0.29389938712120056,
+        0.2742559015750885,
+        0.25545141100883484,
+        0.22576963901519775,
+        0.2079307585954666,
+        0.24914155900478363
+      ],
+      [
+        0.7054738998413086,
+        0.8539307713508606,
+        0.9221710562705994,
+        0.995227575302124,
+        1.0,
+        0.9955264925956726,
+        0.9871774315834045,
+        0.9648488759994507,
+        0.9293156862258911,
+        0.33550918102264404,
+        0.33139121532440186,
+        0.32308056950569153,
+        0.32135188579559326,
+        0.31377658247947693,
+        0.3109651207923889,
+        0.31324508786201477,
+        0.30540403723716736,
+        0.3480979800224304,
+        0.2881461977958679,
+        0.28593939542770386,
+        0.2860979735851288,
+        0.28630706667900085,
+        0.2864839732646942,
+        0.2886858880519867,
+        0.43609726428985596,
+        0.43914493918418884,
+        0.46483513712882996,
+        0.4626930356025696,
+        0.4702044725418091,
+        0.40629586577415466,
+        0.40342196822166443,
+        0.3966600000858307,
+        0.3857835531234741,
+        0.32744163274765015,
+        0.33175063133239746,
+        0.3172719180583954,
+        0.30519741773605347,
+        0.29320383071899414,
+        0.27433881163597107,
+        0.25606462359428406,
+        0.22731678187847137,
+        0.20990385115146637,
+        0.25057658553123474
+      ],
+      [
+        0.6806746125221252,
+        0.8308154940605164,
+        0.9034830331802368,
+        0.9878040552139282,
+        0.9955264925956726,
+        1.0,
+        0.9944660067558289,
+        0.9749224781990051,
+        0.9430608749389648,
+        0.3607690930366516,
+        0.3568904995918274,
+        0.34869030117988586,
+        0.3472835421562195,
+        0.33940771222114563,
+        0.3363325297832489,
+        0.3382657468318939,
+        0.32981014251708984,
+        0.3720569312572479,
+        0.2870592474937439,
+        0.284622460603714,
+        0.28477877378463745,
+        0.2850104570388794,
+        0.2851870656013489,
+        0.28762033581733704,
+        0.4527134895324707,
+        0.4562855660915375,
+        0.4845730662345886,
+        0.4827938675880432,
+        0.4897107779979706,
+        0.42294442653656006,
+        0.4187129735946655,
+        0.41039565205574036,
+        0.39745908975601196,
+        0.3240452706813812,
+        0.32759734988212585,
+        0.3131496012210846,
+        0.30125588178634644,
+        0.2897612750530243,
+        0.2714788019657135,
+        0.2537075877189636,
+        0.22584068775177002,
+        0.20897728204727173,
+        0.24922457337379456
+      ],
+      [
+        0.6546079516410828,
+        0.8077625632286072,
+        0.8851162791252136,
+        0.9771004319190979,
+        0.9871774315834045,
+        0.9944660067558289,
+        1.0,
+        0.9820618033409119,
+        0.953970730304718,
+        0.3724070191383362,
+        0.3690847158432007,
+        0.3610292077064514,
+        0.3599538505077362,
+        0.3517965078353882,
+        0.3483361005783081,
+        0.3496970534324646,
+        0.34060928225517273,
+        0.3831266760826111,
+        0.2930297255516052,
+        0.29048728942871094,
+        0.29062631726264954,
+        0.29089054465293884,
+        0.2910628914833069,
+        0.2935792803764343,
+        0.46357405185699463,
+        0.4668458104133606,
+        0.49484941363334656,
+        0.49225494265556335,
+        0.49875810742378235,
+        0.4283643364906311,
+        0.42346519231796265,
+        0.4143224060535431,
+        0.4004828631877899,
+        0.32099777460098267,
+        0.32415735721588135,
+        0.3096967935562134,
+        0.2979552745819092,
+        0.28647178411483765,
+        0.2684289216995239,
+        0.25093063712120056,
+        0.22366459667682648,
+        0.20713619887828827,
+        0.2471763789653778
+      ],
+      [
+        0.6396195292472839,
+        0.7751748561859131,
+        0.8494495749473572,
+        0.9507975578308105,
+        0.9648488759994507,
+        0.9749224781990051,
+        0.9820618033409119,
+        1.0,
+        0.9852873682975769,
+        0.4005168080329895,
+        0.3977200984954834,
+        0.39006975293159485,
+        0.38888439536094666,
+        0.37970638275146484,
+        0.37458398938179016,
+        0.3730723559856415,
+        0.3639926314353943,
+        0.3965635299682617,
+        0.25505632162094116,
+        0.2525669038295746,
+        0.25264737010002136,
+        0.2528040111064911,
+        0.25296083092689514,
+        0.25539398193359375,
+        0.4444199204444885,
+        0.4489409923553467,
+        0.48802775144577026,
+        0.4915412962436676,
+        0.49782225489616394,
+        0.44944503903388977,
+        0.4436427652835846,
+        0.4329726994037628,
+        0.4182862639427185,
+        0.3326359987258911,
+        0.3348675072193146,
+        0.3216875195503235,
+        0.3106471300125122,
+        0.30101141333580017,
+        0.28571727871894836,
+        0.2689763605594635,
+        0.24261517822742462,
+        0.22421549260616302,
+        0.2646792531013489
+      ],
+      [
+        0.5966438055038452,
+        0.7229080200195312,
+        0.7978618741035461,
+        0.9107967615127563,
+        0.9293156862258911,
+        0.9430608749389648,
+        0.953970730304718,
+        0.9852873682975769,
+        1.0,
+        0.4309193193912506,
+        0.42949405312538147,
+        0.4229198396205902,
+        0.4221244156360626,
+        0.4122016131877899,
+        0.40526336431503296,
+        0.4009464681148529,
+        0.3909369707107544,
+        0.4178644120693207,
+        0.23692750930786133,
+        0.2343255579471588,
+        0.23436222970485687,
+        0.2345062792301178,
+        0.23465289175510406,
+        0.23714682459831238,
+        0.44334208965301514,
+        0.44831061363220215,
+        0.49308687448501587,
+        0.4989806115627289,
+        0.5044687986373901,
+        0.46394869685173035,
+        0.4565954804420471,
+        0.4442926347255707,
+        0.42832520604133606,
+        0.32949790358543396,
+        0.3316981792449951,
+        0.31988200545310974,
+        0.3099154829978943,
+        0.3022046983242035,
+        0.28973516821861267,
+        0.27511394023895264,
+        0.2516450881958008,
+        0.23347976803779602,
+        0.27288132905960083
+      ],
+      [
+        0.085598886013031,
+        0.10106048732995987,
+        0.11317962408065796,
+        0.308550626039505,
+        0.33550918102264404,
+        0.3607690930366516,
+        0.3724070191383362,
+        0.4005168080329895,
+        0.4309193193912506,
+        1.0,
+        0.9983712434768677,
+        0.9967196583747864,
+        0.993431031703949,
+        0.9915507435798645,
+        0.9917429089546204,
+        0.9871540069580078,
+        0.9866191744804382,
+        0.9566135406494141,
+        0.0430295355618,
+        0.03934454545378685,
+        0.03954160958528519,
+        0.03954703360795975,
+        0.03969861567020416,
+        0.043120093643665314,
+        0.5498442053794861,
+        0.568802535533905,
+        0.7249490022659302,
+        0.7664463520050049,
+        0.7588014602661133,
+        0.8236193656921387,
+        0.7813343405723572,
+        0.7313903570175171,
+        0.6704528331756592,
+        0.07240330427885056,
+        0.0693894773721695,
+        0.06366771459579468,
+        0.058279674500226974,
+        0.0554267056286335,
+        0.050616730004549026,
+        0.04563816264271736,
+        0.04066110774874687,
+        0.03758147731423378,
+        0.04408200830221176
+      ],
+      [
+        0.07780761271715164,
+        0.09334167093038559,
+        0.10702235996723175,
+        0.3040868937969208,
+        0.33139121532440186,
+        0.3568904995918274,
+        0.3690847158432007,
+        0.3977200984954834,
+        0.42949405312538147,
+        0.9983712434768677,
+        1.0,
+        0.9991177916526794,
+        0.9967723488807678,
+        0.9949100613594055,
+        0.9938264489173889,
+        0.9891536235809326,
+        0.9876658320426941,
+        0.9608913064002991,
+        0.05062812194228172,
+        0.046665869653224945,
+        0.04688529297709465,
+        0.046895939856767654,
+        0.04706761613488197,
+        0.05072401463985443,
+        0.5625230073928833,
+        0.5815515518188477,
+        0.7339819669723511,
+        0.7737396955490112,
+        0.7656188607215881,
+        0.8210443258285522,
+        0.7790476083755493,
+        0.7293313145637512,
+        0.6685813069343567,
+        0.07231699675321579,
+        0.069251149892807,
+        0.06337570399045944,
+        0.05794829502701759,
+        0.05501179024577141,
+        0.05034567788243294,
+        0.04554736986756325,
+        0.04089762642979622,
+        0.0380101203918457,
+        0.04412412270903587
+      ],
+      [
+        0.07064133882522583,
+        0.08555062115192413,
+        0.09908141940832138,
+        0.29570043087005615,
+        0.32308056950569153,
+        0.34869030117988586,
+        0.3610292077064514,
+        0.39006975293159485,
+        0.4229198396205902,
+        0.9967196583747864,
+        0.9991177916526794,
+        1.0,
+        0.9984608888626099,
+        0.9968641400337219,
+        0.9954223036766052,
+        0.9907151460647583,
+        0.9888972640037537,
+        0.9628637433052063,
+        0.05198612064123154,
+        0.0479416586458683,
+        0.0481654517352581,
+        0.04817423224449158,
+        0.048352621495723724,
+        0.052079249173402786,
+        0.5660314559936523,
+        0.5851565003395081,
+        0.7368158102035522,
+        0.7762715220451355,
+        0.7680290341377258,
+        0.8212220072746277,
+        0.7793737649917603,
+        0.7299240231513977,
+        0.6690068244934082,
+        0.07289182394742966,
+        0.0697130560874939,
+        0.06382288783788681,
+        0.058345895260572433,
+        0.055434055626392365,
+        0.050859484821558,
+        0.04605194926261902,
+        0.04151143878698349,
+        0.03859316557645798,
+        0.04483146220445633
+      ],
+      [
+        0.06587867438793182,
+        0.08125422894954681,
+        0.09638337790966034,
+        0.2935424745082855,
+        0.32135188579559326,
+        0.3472835421562195,
+        0.3599538505077362,
+        0.38888439536094666,
+        0.4221244156360626,
+        0.993431031703949,
+        0.9967723488807678,
+        0.9984608888626099,
+        1.0,
+        0.998919665813446,
+        0.9969019293785095,
+        0.9931889176368713,
+        0.9905962944030762,
+        0.9688683152198792,
+        0.0649133026599884,
+        0.06056395545601845,
+        0.06081191077828407,
+        0.06082715466618538,
+        0.061027247458696365,
+        0.06504278630018234,
+        0.5855360627174377,
+        0.6047874689102173,
+        0.7524403929710388,
+        0.7901456952095032,
+        0.7814924120903015,
+        0.822943925857544,
+        0.7818796634674072,
+        0.7328982949256897,
+        0.6719599366188049,
+        0.07769713550806046,
+        0.07411515712738037,
+        0.06803729385137558,
+        0.06221542879939079,
+        0.05910384654998779,
+        0.054406825453042984,
+        0.0491962805390358,
+        0.04431117698550224,
+        0.04105953127145767,
+        0.047917358577251434
+      ],
+      [
+        0.06444326043128967,
+        0.0780123695731163,
+        0.09181856364011765,
+        0.2864590287208557,
+        0.31377658247947693,
+        0.33940771222114563,
+        0.3517965078353882,
+        0.37970638275146484,
+        0.4122016131877899,
+        0.9915507435798645,
+        0.9949100613594055,
+        0.9968641400337219,
+        0.998919665813446,
+        1.0,
+        0.998546838760376,
+        0.9956219792366028,
+        0.993252158164978,
+        0.972076416015625,
+        0.06579349935054779,
+        0.06141830235719681,
+        0.061670996248722076,
+        0.061693765223026276,
+        0.06189659610390663,
+        0.06596172600984573,
+        0.5887587070465088,
+        0.60823655128479,
+        0.7562814950942993,
+        0.794303297996521,
+        0.7856388092041016,
+        0.8270909786224365,
+        0.7860783934593201,
+        0.7372952699661255,
+        0.6760224103927612,
+        0.08032910525798798,
+        0.07661429792642593,
+        0.07035835832357407,
+        0.06424151360988617,
+        0.06096977740526199,
+        0.055847618728876114,
+        0.050160396844148636,
+        0.0446544885635376,
+        0.04107876121997833,
+        0.04873289167881012
+      ],
+      [
+        0.06620140373706818,
+        0.07894133031368256,
+        0.09098811447620392,
+        0.2842535078525543,
+        0.3109651207923889,
+        0.3363325297832489,
+        0.3483361005783081,
+        0.37458398938179016,
+        0.40526336431503296,
+        0.9917429089546204,
+        0.9938264489173889,
+        0.9954223036766052,
+        0.9969019293785095,
+        0.998546838760376,
+        1.0,
+        0.9978783130645752,
+        0.9963661432266235,
+        0.9736549258232117,
+        0.05974001809954643,
+        0.05549969524145126,
+        0.05575323849916458,
+        0.055778440088033676,
+        0.05597558617591858,
+        0.05994410067796707,
+        0.5834383964538574,
+        0.603299081325531,
+        0.7547523379325867,
+        0.7943676114082336,
+        0.7866061329841614,
+        0.8345628976821899,
+        0.7933462858200073,
+        0.744498074054718,
+        0.6827337741851807,
+        0.08384349197149277,
+        0.08009275794029236,
+        0.0735638365149498,
+        0.06728626787662506,
+        0.06396643072366714,
+        0.05840763822197914,
+        0.0524911992251873,
+        0.04635629802942276,
+        0.042436614632606506,
+        0.05078994855284691
+      ],
+      [
+        0.06643415987491608,
+        0.08002081513404846,
+        0.0925474762916565,
+        0.2867068648338318,
+        0.31324508786201477,
+        0.3382657468318939,
+        0.3496970534324646,
+        0.3730723559856415,
+        0.4009464681148529,
+        0.9871540069580078,
+        0.9891536235809326,
+        0.9907151460647583,
+        0.9931889176368713,
+        0.9956219792366028,
+        0.9978783130645752,
+        1.0,
+        0.9985812902450562,
+        0.9823518395423889,
+        0.07506875693798065,
+        0.07046352326869965,
+        0.07077838480472565,
+        0.07081238180398941,
+        0.07104198634624481,
+        0.07539550215005875,
+        0.6085659861564636,
+        0.6289919018745422,
+        0.7765320539474487,
+        0.815192461013794,
+        0.8073939681053162,
+        0.8424702882766724,
+        0.8021972179412842,
+        0.7543538212776184,
+        0.6925984621047974,
+        0.09298310428857803,
+        0.08874774724245071,
+        0.08172436058521271,
+        0.07478447258472443,
+        0.07101134210824966,
+        0.06470689922571182,
+        0.05802201107144356,
+        0.05082838982343674,
+        0.04616275802254677,
+        0.0556659996509552
+      ],
+      [
+        0.06744755059480667,
+        0.07889489084482193,
+        0.08911333978176117,
+        0.2797296643257141,
+        0.30540403723716736,
+        0.32981014251708984,
+        0.34060928225517273,
+        0.3639926314353943,
+        0.3909369707107544,
+        0.9866191744804382,
+        0.9876658320426941,
+        0.9888972640037537,
+        0.9905962944030762,
+        0.993252158164978,
+        0.9963661432266235,
+        0.9985812902450562,
+        1.0,
+        0.9805582165718079,
+        0.06232975423336029,
+        0.05793860927224159,
+        0.05823370814323425,
+        0.05825956538319588,
+        0.058471664786338806,
+        0.06263023614883423,
+        0.5946943759918213,
+        0.6155142784118652,
+        0.7682710886001587,
+        0.809515118598938,
+        0.8023097515106201,
+        0.8486509919166565,
+        0.8083965182304382,
+        0.7606802582740784,
+        0.6990503668785095,
+        0.09798076003789902,
+        0.09355077892541885,
+        0.0864337757229805,
+        0.07936399430036545,
+        0.07550866156816483,
+        0.06888752430677414,
+        0.06172937527298927,
+        0.053861312568187714,
+        0.0485219843685627,
+        0.0585828460752964
+      ],
+      [
+        0.06798108667135239,
+        0.09398424625396729,
+        0.1197425052523613,
+        0.32049211859703064,
+        0.3480979800224304,
+        0.3720569312572479,
+        0.3831266760826111,
+        0.3965635299682617,
+        0.4178644120693207,
+        0.9566135406494141,
+        0.9608913064002991,
+        0.9628637433052063,
+        0.9688683152198792,
+        0.972076416015625,
+        0.9736549258232117,
+        0.9823518395423889,
+        0.9805582165718079,
+        1.0,
+        0.20967945456504822,
+        0.20398284494876862,
+        0.20447564125061035,
+        0.20453287661075592,
+        0.20486538112163544,
+        0.21031388640403748,
+        0.7319108843803406,
+        0.7499132752418518,
+        0.8658959269523621,
+        0.8875241875648499,
+        0.8780434131622314,
+        0.8381298184394836,
+        0.8010719418525696,
+        0.7571243047714233,
+        0.6966723799705505,
+        0.11264808475971222,
+        0.10858210921287537,
+        0.1002814769744873,
+        0.09172182530164719,
+        0.08617549389600754,
+        0.07838811725378036,
+        0.06973807513713837,
+        0.06031683087348938,
+        0.054660484194755554,
+        0.06589983403682709
+      ],
+      [
+        0.023102257400751114,
+        0.11173419654369354,
+        0.2184595763683319,
+        0.27435117959976196,
+        0.2881461977958679,
+        0.2870592474937439,
+        0.2930297255516052,
+        0.25505632162094116,
+        0.23692750930786133,
+        0.0430295355618,
+        0.05062812194228172,
+        0.05198612064123154,
+        0.0649133026599884,
+        0.06579349935054779,
+        0.05974001809954643,
+        0.07506875693798065,
+        0.06232975423336029,
+        0.20967945456504822,
+        1.0,
+        0.999909520149231,
+        0.9999238848686218,
+        0.999921441078186,
+        0.9999275803565979,
+        0.9999585747718811,
+        0.7119568586349487,
+        0.671326220035553,
+        0.5175796151161194,
+        0.41512802243232727,
+        0.4003653824329376,
+        0.06566229462623596,
+        0.06661772727966309,
+        0.07006547600030899,
+        0.06426654756069183,
+        0.07124065607786179,
+        0.07693160325288773,
+        0.07690145075321198,
+        0.06902671605348587,
+        0.06145131587982178,
+        0.06265770643949509,
+        0.055518437176942825,
+        0.04859182983636856,
+        0.045823391526937485,
+        0.05258903652429581
+      ],
+      [
+        0.02318720705807209,
+        0.11162395775318146,
+        0.2182939201593399,
+        0.27240100502967834,
+        0.28593939542770386,
+        0.284622460603714,
+        0.29048728942871094,
+        0.2525669038295746,
+        0.2343255579471588,
+        0.03934454545378685,
+        0.046665869653224945,
+        0.0479416586458683,
+        0.06056395545601845,
+        0.06141830235719681,
+        0.05549969524145126,
+        0.07046352326869965,
+        0.05793860927224159,
+        0.20398284494876862,
+        0.999909520149231,
+        1.0,
+        0.9999940991401672,
+        0.9999884366989136,
+        0.9999811053276062,
+        0.9998469948768616,
+        0.7045469284057617,
+        0.6635348796844482,
+        0.5103201270103455,
+        0.40810084342956543,
+        0.393448144197464,
+        0.0624198317527771,
+        0.0633949413895607,
+        0.06690343469381332,
+        0.06137187033891678,
+        0.07085655629634857,
+        0.0765998587012291,
+        0.07666971534490585,
+        0.06883713603019714,
+        0.061300039291381836,
+        0.06258251518011093,
+        0.0554729625582695,
+        0.048547327518463135,
+        0.04577719047665596,
+        0.05257975682616234
+      ],
+      [
+        0.023029332980513573,
+        0.11166036128997803,
+        0.21833431720733643,
+        0.27254536747932434,
+        0.2860979735851288,
+        0.28477877378463745,
+        0.29062631726264954,
+        0.25264737010002136,
+        0.23436222970485687,
+        0.03954160958528519,
+        0.04688529297709465,
+        0.0481654517352581,
+        0.06081191077828407,
+        0.061670996248722076,
+        0.05575323849916458,
+        0.07077838480472565,
+        0.05823370814323425,
+        0.20447564125061035,
+        0.9999238848686218,
+        0.9999940991401672,
+        1.0,
+        0.9999904036521912,
+        0.9999901652336121,
+        0.9998791813850403,
+        0.7054231762886047,
+        0.6644848585128784,
+        0.511172354221344,
+        0.40891215205192566,
+        0.3942578136920929,
+        0.06265631318092346,
+        0.06365527212619781,
+        0.06717797368764877,
+        0.06163809821009636,
+        0.07093967497348785,
+        0.07665561139583588,
+        0.07666600495576859,
+        0.06881962716579437,
+        0.0612604096531868,
+        0.06247647851705551,
+        0.05536418780684471,
+        0.04843009635806084,
+        0.045680876821279526,
+        0.05250919237732887
+      ],
+      [
+        0.02303888462483883,
+        0.11173491179943085,
+        0.21851415932178497,
+        0.27274295687675476,
+        0.28630706667900085,
+        0.2850104570388794,
+        0.29089054465293884,
+        0.2528040111064911,
+        0.2345062792301178,
+        0.03954703360795975,
+        0.046895939856767654,
+        0.04817423224449158,
+        0.06082715466618538,
+        0.061693765223026276,
+        0.055778440088033676,
+        0.07081238180398941,
+        0.05825956538319588,
+        0.20453287661075592,
+        0.999921441078186,
+        0.9999884366989136,
+        0.9999904036521912,
+        1.0,
+        0.9999964237213135,
+        0.9998904466629028,
+        0.7056722044944763,
+        0.6647428870201111,
+        0.5114345550537109,
+        0.40917566418647766,
+        0.3945251703262329,
+        0.06278237700462341,
+        0.06377211213111877,
+        0.06728610396385193,
+        0.06173764541745186,
+        0.07106957584619522,
+        0.07678575813770294,
+        0.07677685469388962,
+        0.0689200833439827,
+        0.061353541910648346,
+        0.06254272162914276,
+        0.05543401837348938,
+        0.048503417521715164,
+        0.04575222730636597,
+        0.05261457711458206
+      ],
+      [
+        0.0230090469121933,
+        0.1118139699101448,
+        0.21860040724277496,
+        0.27290886640548706,
+        0.2864839732646942,
+        0.2851870656013489,
+        0.2910628914833069,
+        0.25296083092689514,
+        0.23465289175510406,
+        0.03969861567020416,
+        0.04706761613488197,
+        0.048352621495723724,
+        0.061027247458696365,
+        0.06189659610390663,
+        0.05597558617591858,
+        0.07104198634624481,
+        0.058471664786338806,
+        0.20486538112163544,
+        0.9999275803565979,
+        0.9999811053276062,
+        0.9999901652336121,
+        0.9999964237213135,
+        1.0,
+        0.9999097585678101,
+        0.706282377243042,
+        0.6654007434844971,
+        0.5120447874069214,
+        0.40976452827453613,
+        0.3951128423213959,
+        0.06300974637269974,
+        0.06402121484279633,
+        0.06754946708679199,
+        0.06200111284852028,
+        0.07124228030443192,
+        0.07693175226449966,
+        0.07687943428754807,
+        0.06900914758443832,
+        0.06142721697688103,
+        0.06256409734487534,
+        0.05545356869697571,
+        0.0485139898955822,
+        0.04577886685729027,
+        0.05267048999667168
+      ],
+      [
+        0.023262370377779007,
+        0.11213553696870804,
+        0.21891941130161285,
+        0.2748803198337555,
+        0.2886858880519867,
+        0.28762033581733704,
+        0.2935792803764343,
+        0.25539398193359375,
+        0.23714682459831238,
+        0.043120093643665314,
+        0.05072401463985443,
+        0.052079249173402786,
+        0.06504278630018234,
+        0.06596172600984573,
+        0.05994410067796707,
+        0.07539550215005875,
+        0.06263023614883423,
+        0.21031388640403748,
+        0.9999585747718811,
+        0.9998469948768616,
+        0.9998791813850403,
+        0.9998904466629028,
+        0.9999097585678101,
+        1.0,
+        0.7137615084648132,
+        0.6733015179634094,
+        0.51947420835495,
+        0.4169948399066925,
+        0.40224966406822205,
+        0.0663842037320137,
+        0.06739447265863419,
+        0.07088898122310638,
+        0.06509575247764587,
+        0.0720483735203743,
+        0.07764078676700592,
+        0.07742615044116974,
+        0.069486103951931,
+        0.06184566766023636,
+        0.06280297785997391,
+        0.05563516169786453,
+        0.04863174259662628,
+        0.045876212418079376,
+        0.05289127677679062
+      ],
+      [
+        0.07920751720666885,
+        0.15420915186405182,
+        0.23645025491714478,
+        0.40867719054222107,
+        0.43609726428985596,
+        0.4527134895324707,
+        0.46357405185699463,
+        0.4444199204444885,
+        0.44334208965301514,
+        0.5498442053794861,
+        0.5625230073928833,
+        0.5660314559936523,
+        0.5855360627174377,
+        0.5887587070465088,
+        0.5834383964538574,
+        0.6085659861564636,
+        0.5946943759918213,
+        0.7319108843803406,
+        0.7119568586349487,
+        0.7045469284057617,
+        0.7054231762886047,
+        0.7056722044944763,
+        0.706282377243042,
+        0.7137615084648132,
+        1.0,
+        0.9971895813941956,
+        0.950676441192627,
+        0.8982461094856262,
+        0.8854431509971619,
+        0.5648123621940613,
+        0.552385151386261,
+        0.5359407663345337,
+        0.5004628300666809,
+        0.19620217382907867,
+        0.19262436032295227,
+        0.181674525141716,
+        0.16961324214935303,
+        0.15909402072429657,
+        0.14928825199604034,
+        0.13633903861045837,
+        0.12095215916633606,
+        0.11138390004634857,
+        0.12936289608478546
+      ],
+      [
+        0.08516508340835571,
+        0.15838824212551117,
+        0.23678229749202728,
+        0.41179877519607544,
+        0.43914493918418884,
+        0.4562855660915375,
+        0.4668458104133606,
+        0.4489409923553467,
+        0.44831061363220215,
+        0.568802535533905,
+        0.5815515518188477,
+        0.5851565003395081,
+        0.6047874689102173,
+        0.60823655128479,
+        0.603299081325531,
+        0.6289919018745422,
+        0.6155142784118652,
+        0.7499132752418518,
+        0.671326220035553,
+        0.6635348796844482,
+        0.6644848585128784,
+        0.6647428870201111,
+        0.6654007434844971,
+        0.6733015179634094,
+        0.9971895813941956,
+        1.0,
+        0.9640783667564392,
+        0.9180646538734436,
+        0.9064103364944458,
+        0.5952581167221069,
+        0.583310604095459,
+        0.5669860243797302,
+        0.5310094356536865,
+        0.21623265743255615,
+        0.21143624186515808,
+        0.19894073903560638,
+        0.18640725314617157,
+        0.17529775202274323,
+        0.16363555192947388,
+        0.14969877898693085,
+        0.13267894089221954,
+        0.12187019735574722,
+        0.14145489037036896
+      ],
+      [
+        0.12063281238079071,
+        0.18471214175224304,
+        0.2491903305053711,
+        0.4371880888938904,
+        0.46483513712882996,
+        0.4845730662345886,
+        0.49484941363334656,
+        0.48802775144577026,
+        0.49308687448501587,
+        0.7249490022659302,
+        0.7339819669723511,
+        0.7368158102035522,
+        0.7524403929710388,
+        0.7562814950942993,
+        0.7547523379325867,
+        0.7765320539474487,
+        0.7682710886001587,
+        0.8658959269523621,
+        0.5175796151161194,
+        0.5103201270103455,
+        0.511172354221344,
+        0.5114345550537109,
+        0.5120447874069214,
+        0.51947420835495,
+        0.950676441192627,
+        0.9640783667564392,
+        1.0,
+        0.9872865676879883,
+        0.9804797172546387,
+        0.7754476070404053,
+        0.7595964074134827,
+        0.7371383905410767,
+        0.6947705149650574,
+        0.29692989587783813,
+        0.28856322169303894,
+        0.272670716047287,
+        0.2585139870643616,
+        0.24620142579078674,
+        0.22987234592437744,
+        0.21223331987857819,
+        0.18888315558433533,
+        0.1721278429031372,
+        0.1982099562883377
+      ],
+      [
+        0.13757441937923431,
+        0.1943701058626175,
+        0.2493284046649933,
+        0.43606802821159363,
+        0.4626930356025696,
+        0.4827938675880432,
+        0.49225494265556335,
+        0.4915412962436676,
+        0.4989806115627289,
+        0.7664463520050049,
+        0.7737396955490112,
+        0.7762715220451355,
+        0.7901456952095032,
+        0.794303297996521,
+        0.7943676114082336,
+        0.815192461013794,
+        0.809515118598938,
+        0.8875241875648499,
+        0.41512802243232727,
+        0.40810084342956543,
+        0.40891215205192566,
+        0.40917566418647766,
+        0.40976452827453613,
+        0.4169948399066925,
+        0.8982461094856262,
+        0.9180646538734436,
+        0.9872865676879883,
+        1.0,
+        0.9966458678245544,
+        0.8506045937538147,
+        0.838002622127533,
+        0.8168523907661438,
+        0.7763916850090027,
+        0.36660265922546387,
+        0.3540237545967102,
+        0.33642521500587463,
+        0.32011911273002625,
+        0.3063182234764099,
+        0.28603243827819824,
+        0.26461443305015564,
+        0.23550668358802795,
+        0.21309895813465118,
+        0.24481619894504547
+      ],
+      [
+        0.1504034847021103,
+        0.20828627049922943,
+        0.2616477906703949,
+        0.4444027245044708,
+        0.4702044725418091,
+        0.4897107779979706,
+        0.49875810742378235,
+        0.49782225489616394,
+        0.5044687986373901,
+        0.7588014602661133,
+        0.7656188607215881,
+        0.7680290341377258,
+        0.7814924120903015,
+        0.7856388092041016,
+        0.7866061329841614,
+        0.8073939681053162,
+        0.8023097515106201,
+        0.8780434131622314,
+        0.4003653824329376,
+        0.393448144197464,
+        0.3942578136920929,
+        0.3945251703262329,
+        0.3951128423213959,
+        0.40224966406822205,
+        0.8854431509971619,
+        0.9064103364944458,
+        0.9804797172546387,
+        0.9966458678245544,
+        1.0,
+        0.8653483986854553,
+        0.8540165424346924,
+        0.8343536853790283,
+        0.7951667308807373,
+        0.3973771035671234,
+        0.3850161135196686,
+        0.3663579523563385,
+        0.34976524114608765,
+        0.3352092504501343,
+        0.3136848509311676,
+        0.29112502932548523,
+        0.2594355046749115,
+        0.23437561094760895,
+        0.26727741956710815
+      ],
+      [
+        0.20180721580982208,
+        0.22415931522846222,
+        0.23879699409008026,
+        0.38802891969680786,
+        0.40629586577415466,
+        0.42294442653656006,
+        0.4283643364906311,
+        0.44944503903388977,
+        0.46394869685173035,
+        0.8236193656921387,
+        0.8210443258285522,
+        0.8212220072746277,
+        0.822943925857544,
+        0.8270909786224365,
+        0.8345628976821899,
+        0.8424702882766724,
+        0.8486509919166565,
+        0.8381298184394836,
+        0.06566229462623596,
+        0.0624198317527771,
+        0.06265631318092346,
+        0.06278237700462341,
+        0.06300974637269974,
+        0.0663842037320137,
+        0.5648123621940613,
+        0.5952581167221069,
+        0.7754476070404053,
+        0.8506045937538147,
+        0.8653483986854553,
+        1.0,
+        0.9904011487960815,
+        0.9704439043998718,
+        0.9368601441383362,
+        0.5288958549499512,
+        0.5148767232894897,
+        0.4965519905090332,
+        0.48039594292640686,
+        0.46464771032333374,
+        0.440955251455307,
+        0.4131018817424774,
+        0.37193602323532104,
+        0.3341248333454132,
+        0.37199312448501587
+      ],
+      [
+        0.20416469871997833,
+        0.22729963064193726,
+        0.24320976436138153,
+        0.3861248791217804,
+        0.40342196822166443,
+        0.4187129735946655,
+        0.42346519231796265,
+        0.4436427652835846,
+        0.4565954804420471,
+        0.7813343405723572,
+        0.7790476083755493,
+        0.7793737649917603,
+        0.7818796634674072,
+        0.7860783934593201,
+        0.7933462858200073,
+        0.8021972179412842,
+        0.8083965182304382,
+        0.8010719418525696,
+        0.06661772727966309,
+        0.0633949413895607,
+        0.06365527212619781,
+        0.06377211213111877,
+        0.06402121484279633,
+        0.06739447265863419,
+        0.552385151386261,
+        0.583310604095459,
+        0.7595964074134827,
+        0.838002622127533,
+        0.8540165424346924,
+        0.9904011487960815,
+        1.0,
+        0.9889050722122192,
+        0.9653147459030151,
+        0.6016663312911987,
+        0.5846797823905945,
+        0.5664506554603577,
+        0.5473917722702026,
+        0.5293927788734436,
+        0.5014894604682922,
+        0.46798568964004517,
+        0.4201577603816986,
+        0.3755658268928528,
+        0.417666494846344
+      ],
+      [
+        0.20618851482868195,
+        0.23028919100761414,
+        0.2469138205051422,
+        0.3806435763835907,
+        0.3966600000858307,
+        0.41039565205574036,
+        0.4143224060535431,
+        0.4329726994037628,
+        0.4442926347255707,
+        0.7313903570175171,
+        0.7293313145637512,
+        0.7299240231513977,
+        0.7328982949256897,
+        0.7372952699661255,
+        0.744498074054718,
+        0.7543538212776184,
+        0.7606802582740784,
+        0.7571243047714233,
+        0.07006547600030899,
+        0.06690343469381332,
+        0.06717797368764877,
+        0.06728610396385193,
+        0.06754946708679199,
+        0.07088898122310638,
+        0.5359407663345337,
+        0.5669860243797302,
+        0.7371383905410767,
+        0.8168523907661438,
+        0.8343536853790283,
+        0.9704439043998718,
+        0.9889050722122192,
+        1.0,
+        0.9857767820358276,
+        0.6702643632888794,
+        0.6520861387252808,
+        0.6344103217124939,
+        0.6132138967514038,
+        0.5937173962593079,
+        0.5627182722091675,
+        0.5241591930389404,
+        0.47016438841819763,
+        0.41858911514282227,
+        0.46332573890686035
+      ],
+      [
+        0.2103184014558792,
+        0.23264256119728088,
+        0.24982884526252747,
+        0.3713274300098419,
+        0.3857835531234741,
+        0.39745908975601196,
+        0.4004828631877899,
+        0.4182862639427185,
+        0.42832520604133606,
+        0.6704528331756592,
+        0.6685813069343567,
+        0.6690068244934082,
+        0.6719599366188049,
+        0.6760224103927612,
+        0.6827337741851807,
+        0.6925984621047974,
+        0.6990503668785095,
+        0.6966723799705505,
+        0.06426654756069183,
+        0.06137187033891678,
+        0.06163809821009636,
+        0.06173764541745186,
+        0.06200111284852028,
+        0.06509575247764587,
+        0.5004628300666809,
+        0.5310094356536865,
+        0.6947705149650574,
+        0.7763916850090027,
+        0.7951667308807373,
+        0.9368601441383362,
+        0.9653147459030151,
+        0.9857767820358276,
+        1.0,
+        0.7446660995483398,
+        0.7232252359390259,
+        0.7054296731948853,
+        0.6820340156555176,
+        0.6602522730827332,
+        0.6257666945457458,
+        0.5834745764732361,
+        0.5233609676361084,
+        0.46513792872428894,
+        0.5128670334815979
+      ],
+      [
+        0.276023268699646,
+        0.3063642978668213,
+        0.3279764652252197,
+        0.3279266059398651,
+        0.32744163274765015,
+        0.3240452706813812,
+        0.32099777460098267,
+        0.3326359987258911,
+        0.32949790358543396,
+        0.07240330427885056,
+        0.07231699675321579,
+        0.07289182394742966,
+        0.07769713550806046,
+        0.08032910525798798,
+        0.08384349197149277,
+        0.09298310428857803,
+        0.09798076003789902,
+        0.11264808475971222,
+        0.07124065607786179,
+        0.07085655629634857,
+        0.07093967497348785,
+        0.07106957584619522,
+        0.07124228030443192,
+        0.0720483735203743,
+        0.19620217382907867,
+        0.21623265743255615,
+        0.29692989587783813,
+        0.36660265922546387,
+        0.3973771035671234,
+        0.5288958549499512,
+        0.6016663312911987,
+        0.6702643632888794,
+        0.7446660995483398,
+        1.0,
+        0.9814231395721436,
+        0.9648851156234741,
+        0.9420889019966125,
+        0.918043851852417,
+        0.8781349062919617,
+        0.8248966336250305,
+        0.7464483380317688,
+        0.6632609367370605,
+        0.7183345556259155
+      ],
+      [
+        0.2877357304096222,
+        0.3168011009693146,
+        0.33550044894218445,
+        0.3327653408050537,
+        0.33175063133239746,
+        0.32759734988212585,
+        0.32415735721588135,
+        0.3348675072193146,
+        0.3316981792449951,
+        0.0693894773721695,
+        0.069251149892807,
+        0.0697130560874939,
+        0.07411515712738037,
+        0.07661429792642593,
+        0.08009275794029236,
+        0.08874774724245071,
+        0.09355077892541885,
+        0.10858210921287537,
+        0.07693160325288773,
+        0.0765998587012291,
+        0.07665561139583588,
+        0.07678575813770294,
+        0.07693175226449966,
+        0.07764078676700592,
+        0.19262436032295227,
+        0.21143624186515808,
+        0.28856322169303894,
+        0.3540237545967102,
+        0.3850161135196686,
+        0.5148767232894897,
+        0.5846797823905945,
+        0.6520861387252808,
+        0.7232252359390259,
+        0.9814231395721436,
+        1.0,
+        0.9873993992805481,
+        0.972152054309845,
+        0.9519630670547485,
+        0.916405975818634,
+        0.8626837134361267,
+        0.7842699885368347,
+        0.6978129744529724,
+        0.7440783381462097
+      ],
+      [
+        0.27862244844436646,
+        0.3035929203033447,
+        0.3209203779697418,
+        0.3181270658969879,
+        0.3172719180583954,
+        0.3131496012210846,
+        0.3096967935562134,
+        0.3216875195503235,
+        0.31988200545310974,
+        0.06366771459579468,
+        0.06337570399045944,
+        0.06382288783788681,
+        0.06803729385137558,
+        0.07035835832357407,
+        0.0735638365149498,
+        0.08172436058521271,
+        0.0864337757229805,
+        0.1002814769744873,
+        0.07690145075321198,
+        0.07666971534490585,
+        0.07666600495576859,
+        0.07677685469388962,
+        0.07687943428754807,
+        0.07742615044116974,
+        0.181674525141716,
+        0.19894073903560638,
+        0.272670716047287,
+        0.33642521500587463,
+        0.3663579523563385,
+        0.4965519905090332,
+        0.5664506554603577,
+        0.6344103217124939,
+        0.7054296731948853,
+        0.9648851156234741,
+        0.9873993992805481,
+        1.0,
+        0.9873649477958679,
+        0.9710380434989929,
+        0.9398236274719238,
+        0.8875017166137695,
+        0.8098384737968445,
+        0.7214956879615784,
+        0.7621151208877563
+      ],
+      [
+        0.2714858949184418,
+        0.2928607165813446,
+        0.30974993109703064,
+        0.30604374408721924,
+        0.30519741773605347,
+        0.30125588178634644,
+        0.2979552745819092,
+        0.3106471300125122,
+        0.3099154829978943,
+        0.058279674500226974,
+        0.05794829502701759,
+        0.058345895260572433,
+        0.06221542879939079,
+        0.06424151360988617,
+        0.06728626787662506,
+        0.07478447258472443,
+        0.07936399430036545,
+        0.09172182530164719,
+        0.06902671605348587,
+        0.06883713603019714,
+        0.06881962716579437,
+        0.0689200833439827,
+        0.06900914758443832,
+        0.069486103951931,
+        0.16961324214935303,
+        0.18640725314617157,
+        0.2585139870643616,
+        0.32011911273002625,
+        0.34976524114608765,
+        0.48039594292640686,
+        0.5473917722702026,
+        0.6132138967514038,
+        0.6820340156555176,
+        0.9420889019966125,
+        0.972152054309845,
+        0.9873649477958679,
+        1.0,
+        0.9886766672134399,
+        0.9647974371910095,
+        0.9186524748802185,
+        0.8474200963973999,
+        0.7625328302383423,
+        0.7892231345176697
+      ],
+      [
+        0.2638208270072937,
+        0.28051790595054626,
+        0.2971762716770172,
+        0.29389938712120056,
+        0.29320383071899414,
+        0.2897612750530243,
+        0.28647178411483765,
+        0.30101141333580017,
+        0.3022046983242035,
+        0.0554267056286335,
+        0.05501179024577141,
+        0.055434055626392365,
+        0.05910384654998779,
+        0.06096977740526199,
+        0.06396643072366714,
+        0.07101134210824966,
+        0.07550866156816483,
+        0.08617549389600754,
+        0.06145131587982178,
+        0.061300039291381836,
+        0.0612604096531868,
+        0.061353541910648346,
+        0.06142721697688103,
+        0.06184566766023636,
+        0.15909402072429657,
+        0.17529775202274323,
+        0.24620142579078674,
+        0.3063182234764099,
+        0.3352092504501343,
+        0.46464771032333374,
+        0.5293927788734436,
+        0.5937173962593079,
+        0.6602522730827332,
+        0.918043851852417,
+        0.9519630670547485,
+        0.9710380434989929,
+        0.9886766672134399,
+        1.0,
+        0.9843608736991882,
+        0.9462465643882751,
+        0.8798750638961792,
+        0.7969598770141602,
+        0.8162945508956909
+      ],
+      [
+        0.24908611178398132,
+        0.2587052285671234,
+        0.2757277190685272,
+        0.2742559015750885,
+        0.27433881163597107,
+        0.2714788019657135,
+        0.2684289216995239,
+        0.28571727871894836,
+        0.28973516821861267,
+        0.050616730004549026,
+        0.05034567788243294,
+        0.050859484821558,
+        0.054406825453042984,
+        0.055847618728876114,
+        0.05840763822197914,
+        0.06470689922571182,
+        0.06888752430677414,
+        0.07838811725378036,
+        0.06265770643949509,
+        0.06258251518011093,
+        0.06247647851705551,
+        0.06254272162914276,
+        0.06256409734487534,
+        0.06280297785997391,
+        0.14928825199604034,
+        0.16363555192947388,
+        0.22987234592437744,
+        0.28603243827819824,
+        0.3136848509311676,
+        0.440955251455307,
+        0.5014894604682922,
+        0.5627182722091675,
+        0.6257666945457458,
+        0.8781349062919617,
+        0.916405975818634,
+        0.9398236274719238,
+        0.9647974371910095,
+        0.9843608736991882,
+        1.0,
+        0.9794701933860779,
+        0.9287322163581848,
+        0.8544009327888489,
+        0.8514858484268188
+      ],
+      [
+        0.23528440296649933,
+        0.23799873888492584,
+        0.2561028003692627,
+        0.25545141100883484,
+        0.25606462359428406,
+        0.2537075877189636,
+        0.25093063712120056,
+        0.2689763605594635,
+        0.27511394023895264,
+        0.04563816264271736,
+        0.04554736986756325,
+        0.04605194926261902,
+        0.0491962805390358,
+        0.050160396844148636,
+        0.0524911992251873,
+        0.05802201107144356,
+        0.06172937527298927,
+        0.06973807513713837,
+        0.055518437176942825,
+        0.0554729625582695,
+        0.05536418780684471,
+        0.05543401837348938,
+        0.05545356869697571,
+        0.05563516169786453,
+        0.13633903861045837,
+        0.14969877898693085,
+        0.21223331987857819,
+        0.26461443305015564,
+        0.29112502932548523,
+        0.4131018817424774,
+        0.46798568964004517,
+        0.5241591930389404,
+        0.5834745764732361,
+        0.8248966336250305,
+        0.8626837134361267,
+        0.8875017166137695,
+        0.9186524748802185,
+        0.9462465643882751,
+        0.9794701933860779,
+        1.0,
+        0.9706311821937561,
+        0.9152870774269104,
+        0.887352705001831
+      ],
+      [
+        0.2028241604566574,
+        0.20439808070659637,
+        0.22381751239299774,
+        0.22576963901519775,
+        0.22731678187847137,
+        0.22584068775177002,
+        0.22366459667682648,
+        0.24261517822742462,
+        0.2516450881958008,
+        0.04066110774874687,
+        0.04089762642979622,
+        0.04151143878698349,
+        0.04431117698550224,
+        0.0446544885635376,
+        0.04635629802942276,
+        0.05082838982343674,
+        0.053861312568187714,
+        0.06031683087348938,
+        0.04859182983636856,
+        0.048547327518463135,
+        0.04843009635806084,
+        0.048503417521715164,
+        0.0485139898955822,
+        0.04863174259662628,
+        0.12095215916633606,
+        0.13267894089221954,
+        0.18888315558433533,
+        0.23550668358802795,
+        0.2594355046749115,
+        0.37193602323532104,
+        0.4201577603816986,
+        0.47016438841819763,
+        0.5233609676361084,
+        0.7464483380317688,
+        0.7842699885368347,
+        0.8098384737968445,
+        0.8474200963973999,
+        0.8798750638961792,
+        0.9287322163581848,
+        0.9706311821937561,
+        1.0,
+        0.9748941659927368,
+        0.9173017144203186
+      ],
+      [
+        0.18157896399497986,
+        0.1831863820552826,
+        0.20405621826648712,
+        0.2079307585954666,
+        0.20990385115146637,
+        0.20897728204727173,
+        0.20713619887828827,
+        0.22421549260616302,
+        0.23347976803779602,
+        0.03758147731423378,
+        0.0380101203918457,
+        0.03859316557645798,
+        0.04105953127145767,
+        0.04107876121997833,
+        0.042436614632606506,
+        0.04616275802254677,
+        0.0485219843685627,
+        0.054660484194755554,
+        0.045823391526937485,
+        0.04577719047665596,
+        0.045680876821279526,
+        0.04575222730636597,
+        0.04577886685729027,
+        0.045876212418079376,
+        0.11138390004634857,
+        0.12187019735574722,
+        0.1721278429031372,
+        0.21309895813465118,
+        0.23437561094760895,
+        0.3341248333454132,
+        0.3755658268928528,
+        0.41858911514282227,
+        0.46513792872428894,
+        0.6632609367370605,
+        0.6978129744529724,
+        0.7214956879615784,
+        0.7625328302383423,
+        0.7969598770141602,
+        0.8544009327888489,
+        0.9152870774269104,
+        0.9748941659927368,
+        1.0,
+        0.9208613038063049
+      ],
+      [
+        0.21474751830101013,
+        0.22291426360607147,
+        0.24675846099853516,
+        0.24914155900478363,
+        0.25057658553123474,
+        0.24922457337379456,
+        0.2471763789653778,
+        0.2646792531013489,
+        0.27288132905960083,
+        0.04408200830221176,
+        0.04412412270903587,
+        0.04483146220445633,
+        0.047917358577251434,
+        0.04873289167881012,
+        0.05078994855284691,
+        0.0556659996509552,
+        0.0585828460752964,
+        0.06589983403682709,
+        0.05258903652429581,
+        0.05257975682616234,
+        0.05250919237732887,
+        0.05261457711458206,
+        0.05267048999667168,
+        0.05289127677679062,
+        0.12936289608478546,
+        0.14145489037036896,
+        0.1982099562883377,
+        0.24481619894504547,
+        0.26727741956710815,
+        0.37199312448501587,
+        0.417666494846344,
+        0.46332573890686035,
+        0.5128670334815979,
+        0.7183345556259155,
+        0.7440783381462097,
+        0.7621151208877563,
+        0.7892231345176697,
+        0.8162945508956909,
+        0.8514858484268188,
+        0.887352705001831,
+        0.9173017144203186,
+        0.9208613038063049,
+        1.0
+      ]
+    ]
+  },
+  "logit_lens": {
+    "embedding": {
+      "entropy": 0.008176468312740326,
+      "entropy_std": 0.2515474855899811,
+      "top1_prob": 0.9989781379699707,
+      "correct_rank_mean": 14393.49609375,
+      "correct_rank_median": 10188.0,
+      "log_rank_mean": 8.11260986328125,
+      "agreement_with_final": 0.005598506890237331
+    },
+    "expand_0": {
+      "entropy": 0.034236982464790344,
+      "entropy_std": 0.3024539649486542,
+      "top1_prob": 0.9954218864440918,
+      "correct_rank_mean": 11135.865234375,
+      "correct_rank_median": 5304.0,
+      "log_rank_mean": 7.781420707702637,
+      "agreement_with_final": 0.005598506890237331
+    },
+    "expand_1": {
+      "entropy": 0.14663031697273254,
+      "entropy_std": 0.5081619620323181,
+      "top1_prob": 0.9832598567008972,
+      "correct_rank_mean": 9871.8037109375,
+      "correct_rank_median": 4662.0,
+      "log_rank_mean": 7.68184757232666,
+      "agreement_with_final": 0.005598506890237331
+    },
+    "expand_2": {
+      "entropy": 0.258383572101593,
+      "entropy_std": 0.7484143376350403,
+      "top1_prob": 0.9704828858375549,
+      "correct_rank_mean": 8762.1416015625,
+      "correct_rank_median": 3955.0,
+      "log_rank_mean": 7.566042900085449,
+      "agreement_with_final": 0.005598506890237331
+    },
+    "expand_3": {
+      "entropy": 0.39039328694343567,
+      "entropy_std": 1.0109078884124756,
+      "top1_prob": 0.9554979801177979,
+      "correct_rank_mean": 8682.91796875,
+      "correct_rank_median": 3898.0,
+      "log_rank_mean": 7.5453033447265625,
+      "agreement_with_final": 0.005598506890237331
+    },
+    "expand_4": {
+      "entropy": 0.5588860511779785,
+      "entropy_std": 1.3780310153961182,
+      "top1_prob": 0.9369558691978455,
+      "correct_rank_mean": 8332.0068359375,
+      "correct_rank_median": 3548.0,
+      "log_rank_mean": 7.495856761932373,
+      "agreement_with_final": 0.005598506890237331
+    },
+    "expand_5": {
+      "entropy": 0.6920637488365173,
+      "entropy_std": 1.6069740056991577,
+      "top1_prob": 0.9192118048667908,
+      "correct_rank_mean": 6979.5546875,
+      "correct_rank_median": 2671.0,
+      "log_rank_mean": 7.241157054901123,
+      "agreement_with_final": 0.005598506890237331
+    },
+    "expand_6": {
+      "entropy": 0.7080297470092773,
+      "entropy_std": 1.597432017326355,
+      "top1_prob": 0.9160333275794983,
+      "correct_rank_mean": 6252.78955078125,
+      "correct_rank_median": 2049.0,
+      "log_rank_mean": 7.042272567749023,
+      "agreement_with_final": 0.005687372293323278
+    },
+    "expand_7": {
+      "entropy": 0.7756950855255127,
+      "entropy_std": 1.7049072980880737,
+      "top1_prob": 0.9075745344161987,
+      "correct_rank_mean": 5574.6640625,
+      "correct_rank_median": 1679.0,
+      "log_rank_mean": 6.867434024810791,
+      "agreement_with_final": 0.0056429398246109486
+    },
+    "expand_8": {
+      "entropy": 0.7931565642356873,
+      "entropy_std": 1.7194257974624634,
+      "top1_prob": 0.9033533930778503,
+      "correct_rank_mean": 5238.6875,
+      "correct_rank_median": 1420.0,
+      "log_rank_mean": 6.73607873916626,
+      "agreement_with_final": 0.0056429398246109486
+    },
+    "expand_9": {
+      "entropy": 0.6823284029960632,
+      "entropy_std": 1.5069533586502075,
+      "top1_prob": 0.9121810793876648,
+      "correct_rank_mean": 4099.94580078125,
+      "correct_rank_median": 933.0,
+      "log_rank_mean": 6.390110492706299,
+      "agreement_with_final": 0.005687372293323278
+    },
+    "expand_10": {
+      "entropy": 1.000943899154663,
+      "entropy_std": 1.9695338010787964,
+      "top1_prob": 0.8712934255599976,
+      "correct_rank_mean": 3916.5166015625,
+      "correct_rank_median": 788.0,
+      "log_rank_mean": 6.270185470581055,
+      "agreement_with_final": 0.005998400505632162
+    },
+    "expand_11": {
+      "entropy": 1.2967911958694458,
+      "entropy_std": 2.363511323928833,
+      "top1_prob": 0.8309510946273804,
+      "correct_rank_mean": 3443.058837890625,
+      "correct_rank_median": 588.0,
+      "log_rank_mean": 6.05585241317749,
+      "agreement_with_final": 0.007553541101515293
+    },
+    "expand_12": {
+      "entropy": 1.423232913017273,
+      "entropy_std": 2.4367549419403076,
+      "top1_prob": 0.8082932233810425,
+      "correct_rank_mean": 2777.310546875,
+      "correct_rank_median": 361.0,
+      "log_rank_mean": 5.695545196533203,
+      "agreement_with_final": 0.010574957355856895
+    },
+    "expand_13": {
+      "entropy": 1.6197071075439453,
+      "entropy_std": 2.5692083835601807,
+      "top1_prob": 0.7709041237831116,
+      "correct_rank_mean": 2434.749755859375,
+      "correct_rank_median": 255.0,
+      "log_rank_mean": 5.40345573425293,
+      "agreement_with_final": 0.028125833719968796
+    },
+    "expand_14": {
+      "entropy": 1.7894773483276367,
+      "entropy_std": 2.620293378829956,
+      "top1_prob": 0.7412071228027344,
+      "correct_rank_mean": 2007.8128662109375,
+      "correct_rank_median": 157.0,
+      "log_rank_mean": 5.057971477508545,
+      "agreement_with_final": 0.037901002913713455
+    },
+    "expand_15": {
+      "entropy": 1.943091869354248,
+      "entropy_std": 2.588176727294922,
+      "top1_prob": 0.7119490504264832,
+      "correct_rank_mean": 1823.1900634765625,
+      "correct_rank_median": 122.0,
+      "log_rank_mean": 4.8463592529296875,
+      "agreement_with_final": 0.05087532103061676
+    },
+    "expand_16": {
+      "entropy": 2.052386999130249,
+      "entropy_std": 2.493408441543579,
+      "top1_prob": 0.6834665536880493,
+      "correct_rank_mean": 1669.0242919921875,
+      "correct_rank_median": 94.0,
+      "log_rank_mean": 4.6481218338012695,
+      "agreement_with_final": 0.06638229638338089
+    },
+    "expand_17": {
+      "entropy": 2.4040215015411377,
+      "entropy_std": 2.651726484298706,
+      "top1_prob": 0.6449254751205444,
+      "correct_rank_mean": 1992.696044921875,
+      "correct_rank_median": 104.0,
+      "log_rank_mean": 4.736515522003174,
+      "agreement_with_final": 0.07122544944286346
+    },
+    "expand_18": {
+      "entropy": 2.3906190395355225,
+      "entropy_std": 2.458310604095459,
+      "top1_prob": 0.6318473219871521,
+      "correct_rank_mean": 1239.644775390625,
+      "correct_rank_median": 40.0,
+      "log_rank_mean": 4.058821201324463,
+      "agreement_with_final": 0.09806273877620697
+    },
+    "expand_19": {
+      "entropy": 2.254875659942627,
+      "entropy_std": 2.2049195766448975,
+      "top1_prob": 0.6253980398178101,
+      "correct_rank_mean": 1140.4500732421875,
+      "correct_rank_median": 27.0,
+      "log_rank_mean": 3.7810096740722656,
+      "agreement_with_final": 0.12956544756889343
+    },
+    "middle_0": {
+      "entropy": 2.791367530822754,
+      "entropy_std": 2.4136030673980713,
+      "top1_prob": 0.5617826581001282,
+      "correct_rank_mean": 1371.789306640625,
+      "correct_rank_median": 32.0,
+      "log_rank_mean": 3.917487382888794,
+      "agreement_with_final": 0.12041233479976654
+    },
+    "compress_0": {
+      "entropy": 2.8886218070983887,
+      "entropy_std": 2.3109898567199707,
+      "top1_prob": 0.5394092202186584,
+      "correct_rank_mean": 1147.898193359375,
+      "correct_rank_median": 22.0,
+      "log_rank_mean": 3.6261489391326904,
+      "agreement_with_final": 0.1596907526254654
+    },
+    "compress_1": {
+      "entropy": 2.8037614822387695,
+      "entropy_std": 2.125213623046875,
+      "top1_prob": 0.5311091542243958,
+      "correct_rank_mean": 876.5938720703125,
+      "correct_rank_median": 14.0,
+      "log_rank_mean": 3.2685093879699707,
+      "agreement_with_final": 0.20385675132274628
+    },
+    "compress_2": {
+      "entropy": 2.894321918487549,
+      "entropy_std": 2.1102190017700195,
+      "top1_prob": 0.5155819654464722,
+      "correct_rank_mean": 864.5591430664062,
+      "correct_rank_median": 14.0,
+      "log_rank_mean": 3.2516262531280518,
+      "agreement_with_final": 0.210121750831604
+    },
+    "compress_3": {
+      "entropy": 2.904087781906128,
+      "entropy_std": 2.03000545501709,
+      "top1_prob": 0.5027743577957153,
+      "correct_rank_mean": 747.5247192382812,
+      "correct_rank_median": 12.0,
+      "log_rank_mean": 3.133939743041992,
+      "agreement_with_final": 0.2354038953781128
+    },
+    "compress_4": {
+      "entropy": 2.9448397159576416,
+      "entropy_std": 1.9586139917373657,
+      "top1_prob": 0.486985445022583,
+      "correct_rank_mean": 675.2647094726562,
+      "correct_rank_median": 12.0,
+      "log_rank_mean": 3.057403326034546,
+      "agreement_with_final": 0.2563316524028778
+    },
+    "compress_5": {
+      "entropy": 2.960257053375244,
+      "entropy_std": 1.9112046957015991,
+      "top1_prob": 0.4762323498725891,
+      "correct_rank_mean": 594.5932006835938,
+      "correct_rank_median": 11.0,
+      "log_rank_mean": 2.9580636024475098,
+      "agreement_with_final": 0.27961432933807373
+    },
+    "compress_6": {
+      "entropy": 2.8116822242736816,
+      "entropy_std": 1.8701006174087524,
+      "top1_prob": 0.48950037360191345,
+      "correct_rank_mean": 566.154296875,
+      "correct_rank_median": 9.0,
+      "log_rank_mean": 2.875314950942993,
+      "agreement_with_final": 0.303830087184906
+    },
+    "compress_7": {
+      "entropy": 2.702578067779541,
+      "entropy_std": 1.7729120254516602,
+      "top1_prob": 0.49450957775115967,
+      "correct_rank_mean": 445.0347595214844,
+      "correct_rank_median": 6.0,
+      "log_rank_mean": 2.618129253387451,
+      "agreement_with_final": 0.36670219898223877
+    },
+    "compress_8": {
+      "entropy": 2.738481283187866,
+      "entropy_std": 1.7727619409561157,
+      "top1_prob": 0.4886171817779541,
+      "correct_rank_mean": 403.071044921875,
+      "correct_rank_median": 6.0,
+      "log_rank_mean": 2.525726556777954,
+      "agreement_with_final": 0.39091798663139343
+    },
+    "compress_9": {
+      "entropy": 2.7483277320861816,
+      "entropy_std": 1.758489966392517,
+      "top1_prob": 0.48307204246520996,
+      "correct_rank_mean": 361.2685546875,
+      "correct_rank_median": 5.0,
+      "log_rank_mean": 2.424931764602661,
+      "agreement_with_final": 0.422998309135437
+    },
+    "compress_10": {
+      "entropy": 2.584798812866211,
+      "entropy_std": 1.7651761770248413,
+      "top1_prob": 0.5065273642539978,
+      "correct_rank_mean": 316.74725341796875,
+      "correct_rank_median": 4.0,
+      "log_rank_mean": 2.286303758621216,
+      "agreement_with_final": 0.47702834010124207
+    },
+    "compress_11": {
+      "entropy": 2.4372329711914062,
+      "entropy_std": 1.724508285522461,
+      "top1_prob": 0.5221402645111084,
+      "correct_rank_mean": 286.2889099121094,
+      "correct_rank_median": 4.0,
+      "log_rank_mean": 2.1740314960479736,
+      "agreement_with_final": 0.5079978704452515
+    },
+    "compress_12": {
+      "entropy": 2.286668062210083,
+      "entropy_std": 1.6892642974853516,
+      "top1_prob": 0.5403583645820618,
+      "correct_rank_mean": 260.3778991699219,
+      "correct_rank_median": 3.0,
+      "log_rank_mean": 2.076981782913208,
+      "agreement_with_final": 0.5433217883110046
+    },
+    "compress_13": {
+      "entropy": 2.219756841659546,
+      "entropy_std": 1.6175485849380493,
+      "top1_prob": 0.5423508882522583,
+      "correct_rank_mean": 210.5286102294922,
+      "correct_rank_median": 3.0,
+      "log_rank_mean": 1.9495395421981812,
+      "agreement_with_final": 0.6007731556892395
+    },
+    "compress_14": {
+      "entropy": 2.1078646183013916,
+      "entropy_std": 1.5745396614074707,
+      "top1_prob": 0.5566951632499695,
+      "correct_rank_mean": 181.68710327148438,
+      "correct_rank_median": 3.0,
+      "log_rank_mean": 1.8273409605026245,
+      "agreement_with_final": 0.6587576866149902
+    },
+    "compress_15": {
+      "entropy": 2.1085779666900635,
+      "entropy_std": 1.5215522050857544,
+      "top1_prob": 0.5488401055335999,
+      "correct_rank_mean": 158.09202575683594,
+      "correct_rank_median": 2.0,
+      "log_rank_mean": 1.7517446279525757,
+      "agreement_with_final": 0.6880831718444824
+    },
+    "compress_16": {
+      "entropy": 2.0532541275024414,
+      "entropy_std": 1.4450801610946655,
+      "top1_prob": 0.5455565452575684,
+      "correct_rank_mean": 148.40792846679688,
+      "correct_rank_median": 2.0,
+      "log_rank_mean": 1.7359200716018677,
+      "agreement_with_final": 0.7060783505439758
+    },
+    "compress_17": {
+      "entropy": 2.151466131210327,
+      "entropy_std": 1.4760440587997437,
+      "top1_prob": 0.5291946530342102,
+      "correct_rank_mean": 138.3632354736328,
+      "correct_rank_median": 2.0,
+      "log_rank_mean": 1.7023895978927612,
+      "agreement_with_final": 0.7247844934463501
+    },
+    "compress_18": {
+      "entropy": 2.490158796310425,
+      "entropy_std": 1.6314911842346191,
+      "top1_prob": 0.48671698570251465,
+      "correct_rank_mean": 131.927490234375,
+      "correct_rank_median": 2.0,
+      "log_rank_mean": 1.593283772468567,
+      "agreement_with_final": 0.8141384720802307
+    },
+    "compress_19": {
+      "entropy": 2.896641731262207,
+      "entropy_std": 1.7835479974746704,
+      "top1_prob": 0.4417716860771179,
+      "correct_rank_mean": 120.98338317871094,
+      "correct_rank_median": 2.0,
+      "log_rank_mean": 1.5257868766784668,
+      "agreement_with_final": 1.0
+    },
+    "final_norm": {
+      "entropy": 2.896641731262207,
+      "entropy_std": 1.7835479974746704,
+      "top1_prob": 0.4417716860771179,
+      "correct_rank_mean": 120.98338317871094,
+      "correct_rank_median": 2.0,
+      "log_rank_mean": 1.5257868766784668,
+      "agreement_with_final": 1.0
+    }
+  },
+  "drift": {
+    "expand_0": {
+      "cos_sim_mean": 0.6849911212921143,
+      "cos_sim_std": 0.175172358751297,
+      "l2_distance": 27.544370651245117
+    },
+    "expand_1": {
+      "cos_sim_mean": 0.9739256501197815,
+      "cos_sim_std": 0.019970882683992386,
+      "l2_distance": 8.544537544250488
+    },
+    "expand_2": {
+      "cos_sim_mean": 0.9862149953842163,
+      "cos_sim_std": 0.018325403332710266,
+      "l2_distance": 6.148813247680664
+    },
+    "expand_3": {
+      "cos_sim_mean": 0.9931728839874268,
+      "cos_sim_std": 0.009126277640461922,
+      "l2_distance": 4.215730667114258
+    },
+    "expand_4": {
+      "cos_sim_mean": 0.9931032657623291,
+      "cos_sim_std": 0.008576806634664536,
+      "l2_distance": 4.328185558319092
+    },
+    "expand_5": {
+      "cos_sim_mean": 0.9900708794593811,
+      "cos_sim_std": 0.009801621548831463,
+      "l2_distance": 5.432103633880615
+    },
+    "expand_6": {
+      "cos_sim_mean": 0.985851526260376,
+      "cos_sim_std": 0.011277426965534687,
+      "l2_distance": 6.384981155395508
+    },
+    "expand_7": {
+      "cos_sim_mean": 0.9834489822387695,
+      "cos_sim_std": 0.014593411237001419,
+      "l2_distance": 6.949868679046631
+    },
+    "expand_8": {
+      "cos_sim_mean": 0.9804284572601318,
+      "cos_sim_std": 0.013100787065923214,
+      "l2_distance": 7.284245014190674
+    },
+    "expand_9": {
+      "cos_sim_mean": 0.9666137099266052,
+      "cos_sim_std": 0.03426649048924446,
+      "l2_distance": 7.660122871398926
+    },
+    "expand_10": {
+      "cos_sim_mean": 0.9552152752876282,
+      "cos_sim_std": 0.03713290020823479,
+      "l2_distance": 8.244612693786621
+    },
+    "expand_11": {
+      "cos_sim_mean": 0.9543018341064453,
+      "cos_sim_std": 0.03289920091629028,
+      "l2_distance": 8.666141510009766
+    },
+    "expand_12": {
+      "cos_sim_mean": 0.9556319713592529,
+      "cos_sim_std": 0.029315391555428505,
+      "l2_distance": 8.619108200073242
+    },
+    "expand_13": {
+      "cos_sim_mean": 0.9506610631942749,
+      "cos_sim_std": 0.03158613666892052,
+      "l2_distance": 9.27637004852295
+    },
+    "expand_14": {
+      "cos_sim_mean": 0.950225293636322,
+      "cos_sim_std": 0.024301381781697273,
+      "l2_distance": 9.890229225158691
+    },
+    "expand_15": {
+      "cos_sim_mean": 0.9466367363929749,
+      "cos_sim_std": 0.023674415424466133,
+      "l2_distance": 10.431220054626465
+    },
+    "expand_16": {
+      "cos_sim_mean": 0.9469488263130188,
+      "cos_sim_std": 0.027035264298319817,
+      "l2_distance": 10.755489349365234
+    },
+    "expand_17": {
+      "cos_sim_mean": 0.9249208569526672,
+      "cos_sim_std": 0.025832273066043854,
+      "l2_distance": 18.261241912841797
+    },
+    "expand_18": {
+      "cos_sim_mean": 0.9288403987884521,
+      "cos_sim_std": 0.04455041512846947,
+      "l2_distance": 14.860454559326172
+    },
+    "expand_19": {
+      "cos_sim_mean": 0.9334417581558228,
+      "cos_sim_std": 0.024690203368663788,
+      "l2_distance": 13.679224014282227
+    },
+    "middle_0": {
+      "cos_sim_mean": 0.8935306072235107,
+      "cos_sim_std": 0.030932700261473656,
+      "l2_distance": 18.660053253173828
+    },
+    "compress_0": {
+      "cos_sim_mean": 0.9521554708480835,
+      "cos_sim_std": 0.016932014375925064,
+      "l2_distance": 14.076226234436035
+    },
+    "compress_1": {
+      "cos_sim_mean": 0.9567081332206726,
+      "cos_sim_std": 0.01375015452504158,
+      "l2_distance": 15.834850311279297
+    },
+    "compress_2": {
+      "cos_sim_mean": 0.967133104801178,
+      "cos_sim_std": 0.011383751407265663,
+      "l2_distance": 17.914730072021484
+    },
+    "compress_3": {
+      "cos_sim_mean": 0.9692171216011047,
+      "cos_sim_std": 0.012477421201765537,
+      "l2_distance": 13.46438980102539
+    },
+    "compress_4": {
+      "cos_sim_mean": 0.9710483551025391,
+      "cos_sim_std": 0.010337604209780693,
+      "l2_distance": 14.120949745178223
+    },
+    "compress_5": {
+      "cos_sim_mean": 0.9687831997871399,
+      "cos_sim_std": 0.011583817191421986,
+      "l2_distance": 15.74539852142334
+    },
+    "compress_6": {
+      "cos_sim_mean": 0.9749146103858948,
+      "cos_sim_std": 0.011172610335052013,
+      "l2_distance": 14.680301666259766
+    },
+    "compress_7": {
+      "cos_sim_mean": 0.965900182723999,
+      "cos_sim_std": 0.01336053665727377,
+      "l2_distance": 19.33135223388672
+    },
+    "compress_8": {
+      "cos_sim_mean": 0.9760032892227173,
+      "cos_sim_std": 0.010950364172458649,
+      "l2_distance": 16.689151763916016
+    },
+    "compress_9": {
+      "cos_sim_mean": 0.9721519947052002,
+      "cos_sim_std": 0.01497873105108738,
+      "l2_distance": 19.427019119262695
+    },
+    "compress_10": {
+      "cos_sim_mean": 0.9651305675506592,
+      "cos_sim_std": 0.016658522188663483,
+      "l2_distance": 22.994850158691406
+    },
+    "compress_11": {
+      "cos_sim_mean": 0.9691932201385498,
+      "cos_sim_std": 0.013362148776650429,
+      "l2_distance": 23.314464569091797
+    },
+    "compress_12": {
+      "cos_sim_mean": 0.9701909422874451,
+      "cos_sim_std": 0.012194577604532242,
+      "l2_distance": 23.932886123657227
+    },
+    "compress_13": {
+      "cos_sim_mean": 0.9694362878799438,
+      "cos_sim_std": 0.011056528426706791,
+      "l2_distance": 25.8023624420166
+    },
+    "compress_14": {
+      "cos_sim_mean": 0.9697679281234741,
+      "cos_sim_std": 0.010314267128705978,
+      "l2_distance": 27.431903839111328
+    },
+    "compress_15": {
+      "cos_sim_mean": 0.9717218279838562,
+      "cos_sim_std": 0.01403223630040884,
+      "l2_distance": 26.985631942749023
+    },
+    "compress_16": {
+      "cos_sim_mean": 0.9759195446968079,
+      "cos_sim_std": 0.009822430089116096,
+      "l2_distance": 26.87748908996582
+    },
+    "compress_17": {
+      "cos_sim_mean": 0.97585529088974,
+      "cos_sim_std": 0.01191424485296011,
+      "l2_distance": 27.900794982910156
+    },
+    "compress_18": {
+      "cos_sim_mean": 0.9694885611534119,
+      "cos_sim_std": 0.01752544939517975,
+      "l2_distance": 32.57567596435547
+    },
+    "compress_19": {
+      "cos_sim_mean": 0.9693145155906677,
+      "cos_sim_std": 0.019211014732718468,
+      "l2_distance": 32.98606872558594
+    },
+    "final_norm": {
+      "cos_sim_mean": 0.9926164746284485,
+      "cos_sim_std": 0.004038275685161352,
+      "l2_distance": 86.8482437133789
+    }
+  }
+}

scripts/spectral_output/prisma/results_b.json

@@ -0,0 +1,2844 @@
+{
+  "transformer.wte.weight": {
+    "shape": [
+      50257,
+      1024
+    ],
+    "effective_rank": 925.6200561523438,
+    "stable_rank": 3.222175359725952,
+    "spectral_norm": 462.8680419921875,
+    "frobenius_norm": 830.8675537109375,
+    "mp_bound": 150.2218309787832,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.25901723751664113,
+    "alpha_r2": 0.9398083798430313,
+    "condition_number": 3932.63427734375,
+    "top_10_sv": [
+      462.8680419921875,
+      68.01887512207031,
+      59.577247619628906,
+      55.14397430419922,
+      54.30785369873047,
+      49.892826080322266,
+      49.320430755615234,
+      48.26898956298828,
+      47.46828842163086,
+      46.06719207763672
+    ]
+  },
+  "transformer.wpe.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 151.1204833984375,
+    "stable_rank": 2.6537351608276367,
+    "spectral_norm": 23.605066299438477,
+    "frobenius_norm": 38.45333480834961,
+    "mp_bound": 0.14957082271575928,
+    "n_above_mp": 210,
+    "n_total": 1024,
+    "signal_ratio": 0.205078125,
+    "alpha": 0.718008780427603,
+    "alpha_r2": 0.8641226316938437,
+    "condition_number": 597220.375,
+    "top_10_sv": [
+      23.605066299438477,
+      20.438417434692383,
+      17.248029708862305,
+      8.979466438293457,
+      5.816164970397949,
+      5.2002973556518555,
+      4.532151699066162,
+      3.2729744911193848,
+      2.9242184162139893,
+      2.0841357707977295
+    ]
+  },
+  "transformer.h.0.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 628.722412109375,
+    "stable_rank": 22.675931930541992,
+    "spectral_norm": 62.47161865234375,
+    "frobenius_norm": 297.48516845703125,
+    "mp_bound": 7.63114237099588,
+    "n_above_mp": 243,
+    "n_total": 1024,
+    "signal_ratio": 0.2373046875,
+    "alpha": 0.7995412978432835,
+    "alpha_r2": 0.9108228897637844,
+    "condition_number": 68.46097564697266,
+    "top_10_sv": [
+      62.47161865234375,
+      59.061126708984375,
+      52.00094985961914,
+      47.60552215576172,
+      45.418373107910156,
+      45.035884857177734,
+      44.06340026855469,
+      41.139434814453125,
+      39.21805953979492,
+      38.14364242553711
+    ]
+  },
+  "transformer.h.0.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 737.2996826171875,
+    "stable_rank": 65.48827362060547,
+    "spectral_norm": 5.3187255859375,
+    "frobenius_norm": 43.041690826416016,
+    "mp_bound": 1.4740509986877441,
+    "n_above_mp": 248,
+    "n_total": 1024,
+    "signal_ratio": 0.2421875,
+    "alpha": 0.5869972613657769,
+    "alpha_r2": 0.8128838986239981,
+    "condition_number": 33621.77734375,
+    "top_10_sv": [
+      5.3187255859375,
+      4.55604887008667,
+      4.446892261505127,
+      4.300610065460205,
+      4.185939311981201,
+      4.152495384216309,
+      4.086340427398682,
+      4.0273637771606445,
+      3.9726521968841553,
+      3.915243148803711
+    ]
+  },
+  "transformer.h.0.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 958.4553833007812,
+    "stable_rank": 35.955875396728516,
+    "spectral_norm": 34.83731460571289,
+    "frobenius_norm": 208.895751953125,
+    "mp_bound": 20.392281532287598,
+    "n_above_mp": 2,
+    "n_total": 1024,
+    "signal_ratio": 0.001953125,
+    "alpha": 0.24270056604789897,
+    "alpha_r2": 0.8867966791819655,
+    "condition_number": 47.199832916259766,
+    "top_10_sv": [
+      34.83731460571289,
+      22.28569221496582,
+      20.11617088317871,
+      19.44432830810547,
+      16.447628021240234,
+      15.993996620178223,
+      15.006787300109863,
+      14.720680236816406,
+      14.216849327087402,
+      13.424826622009277
+    ]
+  },
+  "transformer.h.0.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 928.1140747070312,
+    "stable_rank": 34.63430404663086,
+    "spectral_norm": 28.995763778686523,
+    "frobenius_norm": 170.64273071289062,
+    "mp_bound": 14.264991760253906,
+    "n_above_mp": 5,
+    "n_total": 1024,
+    "signal_ratio": 0.0048828125,
+    "alpha": 0.288666051511119,
+    "alpha_r2": 0.8199686145081493,
+    "condition_number": 42.9376106262207,
+    "top_10_sv": [
+      28.995763778686523,
+      17.847505569458008,
+      17.05669403076172,
+      15.134307861328125,
+      14.533385276794434,
+      12.990875244140625,
+      11.749531745910645,
+      11.249499320983887,
+      10.62743091583252,
+      10.352800369262695
+    ]
+  },
+  "transformer.h.1.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 820.3770751953125,
+    "stable_rank": 41.79490661621094,
+    "spectral_norm": 27.69493865966797,
+    "frobenius_norm": 179.04495239257812,
+    "mp_bound": 9.096073287364284,
+    "n_above_mp": 84,
+    "n_total": 1024,
+    "signal_ratio": 0.08203125,
+    "alpha": 0.5097652395048509,
+    "alpha_r2": 0.8774686180910731,
+    "condition_number": 27.94350814819336,
+    "top_10_sv": [
+      27.69493865966797,
+      26.477949142456055,
+      24.700719833374023,
+      22.73528480529785,
+      21.188859939575195,
+      20.210037231445312,
+      19.647275924682617,
+      18.7845458984375,
+      18.384246826171875,
+      17.731191635131836
+    ]
+  },
+  "transformer.h.1.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 609.0930786132812,
+    "stable_rank": 6.79005241394043,
+    "spectral_norm": 35.383201599121094,
+    "frobenius_norm": 92.2005844116211,
+    "mp_bound": 1.7873764038085938,
+    "n_above_mp": 349,
+    "n_total": 1024,
+    "signal_ratio": 0.3408203125,
+    "alpha": 0.7416108698642246,
+    "alpha_r2": 0.8534609001933348,
+    "condition_number": 36300.8515625,
+    "top_10_sv": [
+      35.383201599121094,
+      30.46486473083496,
+      23.07978630065918,
+      17.489381790161133,
+      15.965164184570312,
+      13.38463306427002,
+      11.213841438293457,
+      10.809903144836426,
+      7.931944370269775,
+      7.714986801147461
+    ]
+  },
+  "transformer.h.1.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 952.1654663085938,
+    "stable_rank": 15.651745796203613,
+    "spectral_norm": 54.71965026855469,
+    "frobenius_norm": 216.48345947265625,
+    "mp_bound": 20.210805416107178,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.28617357150294975,
+    "alpha_r2": 0.944003325737223,
+    "condition_number": 25.737693786621094,
+    "top_10_sv": [
+      54.71965026855469,
+      20.120031356811523,
+      18.258190155029297,
+      17.045495986938477,
+      16.402690887451172,
+      16.293865203857422,
+      15.51065731048584,
+      15.293896675109863,
+      14.751503944396973,
+      14.567024230957031
+    ]
+  },
+  "transformer.h.1.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 951.001220703125,
+    "stable_rank": 15.77647876739502,
+    "spectral_norm": 40.80717086791992,
+    "frobenius_norm": 162.0845184326172,
+    "mp_bound": 15.43984866142273,
+    "n_above_mp": 5,
+    "n_total": 1024,
+    "signal_ratio": 0.0048828125,
+    "alpha": 0.27510892798744624,
+    "alpha_r2": 0.9524180259003465,
+    "condition_number": 47.46715545654297,
+    "top_10_sv": [
+      40.80717086791992,
+      25.39755630493164,
+      24.217174530029297,
+      18.221494674682617,
+      15.662391662597656,
+      13.956523895263672,
+      13.657442092895508,
+      13.022187232971191,
+      12.348648071289062,
+      10.786282539367676
+    ]
+  },
+  "transformer.h.2.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 842.8485107421875,
+    "stable_rank": 39.29759216308594,
+    "spectral_norm": 32.41156768798828,
+    "frobenius_norm": 203.18096923828125,
+    "mp_bound": 11.375351081125348,
+    "n_above_mp": 52,
+    "n_total": 1024,
+    "signal_ratio": 0.05078125,
+    "alpha": 0.4740024292360243,
+    "alpha_r2": 0.8802370281377663,
+    "condition_number": 52.709171295166016,
+    "top_10_sv": [
+      32.41156768798828,
+      30.25526237487793,
+      25.755977630615234,
+      24.045995712280273,
+      22.17633819580078,
+      21.890745162963867,
+      21.64605712890625,
+      21.57254981994629,
+      19.244956970214844,
+      18.357177734375
+    ]
+  },
+  "transformer.h.2.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 715.241943359375,
+    "stable_rank": 9.787028312683105,
+    "spectral_norm": 23.163925170898438,
+    "frobenius_norm": 72.466552734375,
+    "mp_bound": 2.168848752975464,
+    "n_above_mp": 315,
+    "n_total": 1024,
+    "signal_ratio": 0.3076171875,
+    "alpha": 0.6080837150928929,
+    "alpha_r2": 0.7822885769468158,
+    "condition_number": 187718.40625,
+    "top_10_sv": [
+      23.163925170898438,
+      9.864102363586426,
+      8.651338577270508,
+      8.504467964172363,
+      7.996184349060059,
+      7.292422294616699,
+      6.915347576141357,
+      6.678709030151367,
+      6.435945510864258,
+      6.021938323974609
+    ]
+  },
+  "transformer.h.2.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 950.7151489257812,
+    "stable_rank": 23.699460983276367,
+    "spectral_norm": 44.83905029296875,
+    "frobenius_norm": 218.28587341308594,
+    "mp_bound": 20.38237237930298,
+    "n_above_mp": 3,
+    "n_total": 1024,
+    "signal_ratio": 0.0029296875,
+    "alpha": 0.2912721396169389,
+    "alpha_r2": 0.9537768474757292,
+    "condition_number": 36.274898529052734,
+    "top_10_sv": [
+      44.83905029296875,
+      22.49462127685547,
+      20.47141456604004,
+      19.455570220947266,
+      18.79952621459961,
+      18.357946395874023,
+      17.340517044067383,
+      16.285560607910156,
+      15.497901916503906,
+      15.356476783752441
+    ]
+  },
+  "transformer.h.2.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 950.6380615234375,
+    "stable_rank": 20.027393341064453,
+    "spectral_norm": 37.39990997314453,
+    "frobenius_norm": 167.3719940185547,
+    "mp_bound": 15.765140533447266,
+    "n_above_mp": 3,
+    "n_total": 1024,
+    "signal_ratio": 0.0029296875,
+    "alpha": 0.2643769291286133,
+    "alpha_r2": 0.9029311778240089,
+    "condition_number": 100.74349212646484,
+    "top_10_sv": [
+      37.39990997314453,
+      23.12250328063965,
+      20.761938095092773,
+      12.250714302062988,
+      10.560304641723633,
+      10.4915771484375,
+      10.370850563049316,
+      10.147944450378418,
+      9.839871406555176,
+      9.63220500946045
+    ]
+  },
+  "transformer.h.3.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 868.9850463867188,
+    "stable_rank": 41.50836181640625,
+    "spectral_norm": 33.2469596862793,
+    "frobenius_norm": 214.20013427734375,
+    "mp_bound": 13.617692360816509,
+    "n_above_mp": 30,
+    "n_total": 1024,
+    "signal_ratio": 0.029296875,
+    "alpha": 0.43598154200527217,
+    "alpha_r2": 0.9121806812022613,
+    "condition_number": 91.30290985107422,
+    "top_10_sv": [
+      33.2469596862793,
+      26.513952255249023,
+      25.003625869750977,
+      23.534202575683594,
+      22.926427841186523,
+      21.938570022583008,
+      21.85960578918457,
+      20.899417877197266,
+      20.27928924560547,
+      19.71728515625
+    ]
+  },
+  "transformer.h.3.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 745.2302856445312,
+    "stable_rank": 16.78188133239746,
+    "spectral_norm": 17.08546257019043,
+    "frobenius_norm": 69.99178314208984,
+    "mp_bound": 2.3438045978546143,
+    "n_above_mp": 303,
+    "n_total": 1024,
+    "signal_ratio": 0.2958984375,
+    "alpha": 0.5602377771042083,
+    "alpha_r2": 0.7441943879202231,
+    "condition_number": 26872.6328125,
+    "top_10_sv": [
+      17.08546257019043,
+      7.187192440032959,
+      6.205941677093506,
+      5.882521629333496,
+      5.645092964172363,
+      5.5240044593811035,
+      5.3767547607421875,
+      5.196966171264648,
+      5.166182994842529,
+      5.101844787597656
+    ]
+  },
+  "transformer.h.3.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 927.2161254882812,
+    "stable_rank": 26.023406982421875,
+    "spectral_norm": 43.73242950439453,
+    "frobenius_norm": 223.09286499023438,
+    "mp_bound": 19.039398908615112,
+    "n_above_mp": 6,
+    "n_total": 1024,
+    "signal_ratio": 0.005859375,
+    "alpha": 0.33058612233889434,
+    "alpha_r2": 0.9492689266361353,
+    "condition_number": 47.069427490234375,
+    "top_10_sv": [
+      43.73242950439453,
+      37.90205764770508,
+      20.579469680786133,
+      19.914993286132812,
+      19.66720199584961,
+      19.149551391601562,
+      18.675325393676758,
+      18.34368896484375,
+      17.572053909301758,
+      17.139053344726562
+    ]
+  },
+  "transformer.h.3.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 938.9752197265625,
+    "stable_rank": 12.513044357299805,
+    "spectral_norm": 49.337677001953125,
+    "frobenius_norm": 174.52601623535156,
+    "mp_bound": 15.4556565284729,
+    "n_above_mp": 2,
+    "n_total": 1024,
+    "signal_ratio": 0.001953125,
+    "alpha": 0.2906680436450817,
+    "alpha_r2": 0.9012577872907297,
+    "condition_number": 150.67166137695312,
+    "top_10_sv": [
+      49.337677001953125,
+      16.639305114746094,
+      12.377195358276367,
+      12.035073280334473,
+      11.454721450805664,
+      11.364490509033203,
+      10.909409523010254,
+      10.634885787963867,
+      10.58263111114502,
+      10.402506828308105
+    ]
+  },
+  "transformer.h.4.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 883.5626831054688,
+    "stable_rank": 66.41778564453125,
+    "spectral_norm": 26.355398178100586,
+    "frobenius_norm": 214.7888641357422,
+    "mp_bound": 14.6059706629891,
+    "n_above_mp": 28,
+    "n_total": 1024,
+    "signal_ratio": 0.02734375,
+    "alpha": 0.41387379570841165,
+    "alpha_r2": 0.928107216532441,
+    "condition_number": 54.06818771362305,
+    "top_10_sv": [
+      26.355398178100586,
+      24.099023818969727,
+      23.613140106201172,
+      22.856849670410156,
+      22.657690048217773,
+      22.315385818481445,
+      21.944866180419922,
+      20.979703903198242,
+      20.732112884521484,
+      20.403736114501953
+    ]
+  },
+  "transformer.h.4.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 744.8408203125,
+    "stable_rank": 14.994495391845703,
+    "spectral_norm": 18.674596786499023,
+    "frobenius_norm": 72.3131332397461,
+    "mp_bound": 2.412200927734375,
+    "n_above_mp": 301,
+    "n_total": 1024,
+    "signal_ratio": 0.2939453125,
+    "alpha": 0.557779475735886,
+    "alpha_r2": 0.7426996015149363,
+    "condition_number": 38687.66015625,
+    "top_10_sv": [
+      18.674596786499023,
+      6.152257442474365,
+      6.06334924697876,
+      5.751345634460449,
+      5.698556423187256,
+      5.618075847625732,
+      5.548093795776367,
+      5.433638572692871,
+      5.330124855041504,
+      5.277598857879639
+    ]
+  },
+  "transformer.h.4.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 928.9017333984375,
+    "stable_rank": 27.40129280090332,
+    "spectral_norm": 42.51769256591797,
+    "frobenius_norm": 222.56414794921875,
+    "mp_bound": 19.081247806549072,
+    "n_above_mp": 3,
+    "n_total": 1024,
+    "signal_ratio": 0.0029296875,
+    "alpha": 0.33154839758470445,
+    "alpha_r2": 0.9484103079592642,
+    "condition_number": 44.41904067993164,
+    "top_10_sv": [
+      42.51769256591797,
+      33.62040710449219,
+      20.00185775756836,
+      19.004039764404297,
+      18.752685546875,
+      18.06087875366211,
+      17.665428161621094,
+      17.001155853271484,
+      16.629446029663086,
+      16.390981674194336
+    ]
+  },
+  "transformer.h.4.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 937.2068481445312,
+    "stable_rank": 12.849010467529297,
+    "spectral_norm": 48.77532958984375,
+    "frobenius_norm": 174.8376922607422,
+    "mp_bound": 15.396579265594482,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.3018330337436229,
+    "alpha_r2": 0.9230008315412146,
+    "condition_number": 90.40140533447266,
+    "top_10_sv": [
+      48.77532958984375,
+      14.544754981994629,
+      14.092583656311035,
+      13.27783489227295,
+      12.841653823852539,
+      12.547450065612793,
+      12.23483657836914,
+      11.652227401733398,
+      11.583452224731445,
+      11.510062217712402
+    ]
+  },
+  "transformer.h.5.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 896.8251953125,
+    "stable_rank": 88.8460693359375,
+    "spectral_norm": 22.261667251586914,
+    "frobenius_norm": 209.83444213867188,
+    "mp_bound": 14.939659960070664,
+    "n_above_mp": 23,
+    "n_total": 1024,
+    "signal_ratio": 0.0224609375,
+    "alpha": 0.39352495861912024,
+    "alpha_r2": 0.9197388047187437,
+    "condition_number": 35.448787689208984,
+    "top_10_sv": [
+      22.261667251586914,
+      20.855236053466797,
+      20.020145416259766,
+      19.449405670166016,
+      19.232900619506836,
+      18.622623443603516,
+      18.29434585571289,
+      17.920202255249023,
+      17.658803939819336,
+      17.550424575805664
+    ]
+  },
+  "transformer.h.5.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 746.4803466796875,
+    "stable_rank": 16.239879608154297,
+    "spectral_norm": 19.48035430908203,
+    "frobenius_norm": 78.50335693359375,
+    "mp_bound": 2.632415771484375,
+    "n_above_mp": 307,
+    "n_total": 1024,
+    "signal_ratio": 0.2998046875,
+    "alpha": 0.5490623315523016,
+    "alpha_r2": 0.7276577467035443,
+    "condition_number": 42145.03515625,
+    "top_10_sv": [
+      19.48035430908203,
+      6.621999740600586,
+      6.383045196533203,
+      6.287996292114258,
+      6.164387226104736,
+      5.886140823364258,
+      5.757631778717041,
+      5.754157066345215,
+      5.641662120819092,
+      5.541800498962402
+    ]
+  },
+  "transformer.h.5.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 936.9888916015625,
+    "stable_rank": 31.094371795654297,
+    "spectral_norm": 39.32979202270508,
+    "frobenius_norm": 219.31207275390625,
+    "mp_bound": 19.247617721557617,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.3193148230456383,
+    "alpha_r2": 0.9361649517765219,
+    "condition_number": 41.62086868286133,
+    "top_10_sv": [
+      39.32979202270508,
+      19.010740280151367,
+      18.782209396362305,
+      17.947439193725586,
+      17.337230682373047,
+      16.85923194885254,
+      16.667734146118164,
+      15.952093124389648,
+      15.685457229614258,
+      15.613792419433594
+    ]
+  },
+  "transformer.h.5.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 936.8583984375,
+    "stable_rank": 15.319039344787598,
+    "spectral_norm": 44.49594497680664,
+    "frobenius_norm": 174.1551055908203,
+    "mp_bound": 15.25501549243927,
+    "n_above_mp": 3,
+    "n_total": 1024,
+    "signal_ratio": 0.0029296875,
+    "alpha": 0.30940596098305206,
+    "alpha_r2": 0.9306552491126057,
+    "condition_number": 69.38197326660156,
+    "top_10_sv": [
+      44.49594497680664,
+      16.51666831970215,
+      16.053695678710938,
+      14.54151725769043,
+      13.180120468139648,
+      12.70102596282959,
+      12.365108489990234,
+      12.008030891418457,
+      11.90465259552002,
+      11.715628623962402
+    ]
+  },
+  "transformer.h.6.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 897.4365844726562,
+    "stable_rank": 115.99281311035156,
+    "spectral_norm": 18.230764389038086,
+    "frobenius_norm": 196.3452606201172,
+    "mp_bound": 13.862052830560662,
+    "n_above_mp": 17,
+    "n_total": 1024,
+    "signal_ratio": 0.0166015625,
+    "alpha": 0.3909539490685798,
+    "alpha_r2": 0.8932138219176853,
+    "condition_number": 32.950740814208984,
+    "top_10_sv": [
+      18.230764389038086,
+      17.658872604370117,
+      16.965496063232422,
+      16.603540420532227,
+      16.098522186279297,
+      15.950654983520508,
+      15.425930976867676,
+      15.198224067687988,
+      14.950895309448242,
+      14.900473594665527
+    ]
+  },
+  "transformer.h.6.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 741.775634765625,
+    "stable_rank": 23.221019744873047,
+    "spectral_norm": 17.442646026611328,
+    "frobenius_norm": 84.05296325683594,
+    "mp_bound": 2.810410976409912,
+    "n_above_mp": 295,
+    "n_total": 1024,
+    "signal_ratio": 0.2880859375,
+    "alpha": 0.5652261405393998,
+    "alpha_r2": 0.7572962195682047,
+    "condition_number": 14903.8291015625,
+    "top_10_sv": [
+      17.442646026611328,
+      8.159473419189453,
+      7.555866241455078,
+      7.282261848449707,
+      7.234253883361816,
+      7.103213787078857,
+      6.917233943939209,
+      6.809220790863037,
+      6.620255470275879,
+      6.525652885437012
+    ]
+  },
+  "transformer.h.6.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 934.8484497070312,
+    "stable_rank": 30.034225463867188,
+    "spectral_norm": 39.7269172668457,
+    "frobenius_norm": 217.7173614501953,
+    "mp_bound": 19.036645889282227,
+    "n_above_mp": 3,
+    "n_total": 1024,
+    "signal_ratio": 0.0029296875,
+    "alpha": 0.32115342994248447,
+    "alpha_r2": 0.9391101648886675,
+    "condition_number": 59.284549713134766,
+    "top_10_sv": [
+      39.7269172668457,
+      20.441776275634766,
+      19.18965721130371,
+      18.47493553161621,
+      17.703449249267578,
+      16.833940505981445,
+      16.46803855895996,
+      15.968595504760742,
+      15.606867790222168,
+      15.231405258178711
+    ]
+  },
+  "transformer.h.6.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 939.2515258789062,
+    "stable_rank": 24.95199203491211,
+    "spectral_norm": 34.625038146972656,
+    "frobenius_norm": 172.95889282226562,
+    "mp_bound": 15.321847558021545,
+    "n_above_mp": 2,
+    "n_total": 1024,
+    "signal_ratio": 0.001953125,
+    "alpha": 0.30995776991553114,
+    "alpha_r2": 0.9308628488187047,
+    "condition_number": 56.80348587036133,
+    "top_10_sv": [
+      34.625038146972656,
+      16.99053382873535,
+      15.084966659545898,
+      13.3975191116333,
+      13.283339500427246,
+      12.721936225891113,
+      12.279509544372559,
+      12.057058334350586,
+      11.876668930053711,
+      11.697145462036133
+    ]
+  },
+  "transformer.h.7.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 910.6389770507812,
+    "stable_rank": 113.14727783203125,
+    "spectral_norm": 17.927513122558594,
+    "frobenius_norm": 190.69622802734375,
+    "mp_bound": 14.044904500546567,
+    "n_above_mp": 8,
+    "n_total": 1024,
+    "signal_ratio": 0.0078125,
+    "alpha": 0.36309653033535555,
+    "alpha_r2": 0.87929726729129,
+    "condition_number": 31.099102020263672,
+    "top_10_sv": [
+      17.927513122558594,
+      17.431472778320312,
+      16.597808837890625,
+      15.969321250915527,
+      15.868179321289062,
+      14.81343936920166,
+      14.50455379486084,
+      14.174510955810547,
+      13.997900009155273,
+      13.888890266418457
+    ]
+  },
+  "transformer.h.7.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 743.44921875,
+    "stable_rank": 27.847932815551758,
+    "spectral_norm": 16.62673568725586,
+    "frobenius_norm": 87.74118041992188,
+    "mp_bound": 2.8953471183776855,
+    "n_above_mp": 311,
+    "n_total": 1024,
+    "signal_ratio": 0.3037109375,
+    "alpha": 0.5651577772345532,
+    "alpha_r2": 0.7372368922623682,
+    "condition_number": 9111.541015625,
+    "top_10_sv": [
+      16.62673568725586,
+      7.667027473449707,
+      7.169197082519531,
+      6.978420257568359,
+      6.833680152893066,
+      6.656124591827393,
+      6.566739082336426,
+      6.528911590576172,
+      6.47317361831665,
+      6.427671432495117
+    ]
+  },
+  "transformer.h.7.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 939.2886352539062,
+    "stable_rank": 31.832117080688477,
+    "spectral_norm": 38.28334045410156,
+    "frobenius_norm": 215.99444580078125,
+    "mp_bound": 19.122761964797974,
+    "n_above_mp": 2,
+    "n_total": 1024,
+    "signal_ratio": 0.001953125,
+    "alpha": 0.31228344301482186,
+    "alpha_r2": 0.9328367415414192,
+    "condition_number": 34.72146224975586,
+    "top_10_sv": [
+      38.28334045410156,
+      19.85346031188965,
+      18.42683219909668,
+      17.141801834106445,
+      16.43387222290039,
+      16.275760650634766,
+      16.053062438964844,
+      15.724404335021973,
+      15.451314926147461,
+      15.216456413269043
+    ]
+  },
+  "transformer.h.7.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 940.928955078125,
+    "stable_rank": 33.200374603271484,
+    "spectral_norm": 31.440378189086914,
+    "frobenius_norm": 181.15872192382812,
+    "mp_bound": 16.195263862609863,
+    "n_above_mp": 3,
+    "n_total": 1024,
+    "signal_ratio": 0.0029296875,
+    "alpha": 0.30703067706661946,
+    "alpha_r2": 0.9333060712253897,
+    "condition_number": 61.141605377197266,
+    "top_10_sv": [
+      31.440378189086914,
+      18.50613784790039,
+      16.681230545043945,
+      15.740647315979004,
+      13.607156753540039,
+      13.430535316467285,
+      13.093123435974121,
+      12.78380298614502,
+      12.500567436218262,
+      12.175090789794922
+    ]
+  },
+  "transformer.h.8.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 905.0512084960938,
+    "stable_rank": 99.04619598388672,
+    "spectral_norm": 19.30182647705078,
+    "frobenius_norm": 192.09555053710938,
+    "mp_bound": 13.908594893171308,
+    "n_above_mp": 12,
+    "n_total": 1024,
+    "signal_ratio": 0.01171875,
+    "alpha": 0.37568865395886447,
+    "alpha_r2": 0.8932899012135317,
+    "condition_number": 26.482255935668945,
+    "top_10_sv": [
+      19.30182647705078,
+      18.0625,
+      17.029926300048828,
+      16.608489990234375,
+      15.877984046936035,
+      15.367476463317871,
+      15.020094871520996,
+      14.934671401977539,
+      14.854327201843262,
+      14.298294067382812
+    ]
+  },
+  "transformer.h.8.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 730.39599609375,
+    "stable_rank": 23.92629051208496,
+    "spectral_norm": 18.780719757080078,
+    "frobenius_norm": 91.8649673461914,
+    "mp_bound": 2.854386568069458,
+    "n_above_mp": 321,
+    "n_total": 1024,
+    "signal_ratio": 0.3134765625,
+    "alpha": 0.590965763955716,
+    "alpha_r2": 0.7524001510220397,
+    "condition_number": 10199.13671875,
+    "top_10_sv": [
+      18.780719757080078,
+      9.103902816772461,
+      8.50258731842041,
+      7.891252517700195,
+      7.823528289794922,
+      7.506181716918945,
+      7.408065319061279,
+      7.162362575531006,
+      7.139276504516602,
+      7.067962646484375
+    ]
+  },
+  "transformer.h.8.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 936.2421875,
+    "stable_rank": 38.09269714355469,
+    "spectral_norm": 34.884918212890625,
+    "frobenius_norm": 215.3072052001953,
+    "mp_bound": 18.746233463287354,
+    "n_above_mp": 2,
+    "n_total": 1024,
+    "signal_ratio": 0.001953125,
+    "alpha": 0.32267201724432737,
+    "alpha_r2": 0.9352114281367789,
+    "condition_number": 38.3657341003418,
+    "top_10_sv": [
+      34.884918212890625,
+      19.184831619262695,
+      18.076223373413086,
+      17.63890266418457,
+      16.88092613220215,
+      16.76192855834961,
+      16.45589828491211,
+      16.098548889160156,
+      15.723489761352539,
+      15.483587265014648
+    ]
+  },
+  "transformer.h.8.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 950.4535522460938,
+    "stable_rank": 39.308773040771484,
+    "spectral_norm": 27.4515380859375,
+    "frobenius_norm": 172.1121063232422,
+    "mp_bound": 15.766410827636719,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.28704826650262233,
+    "alpha_r2": 0.905894394932516,
+    "condition_number": 47.98366165161133,
+    "top_10_sv": [
+      27.4515380859375,
+      14.21110725402832,
+      12.706835746765137,
+      12.113818168640137,
+      11.690884590148926,
+      11.509939193725586,
+      11.169968605041504,
+      11.008914947509766,
+      10.82995319366455,
+      10.744305610656738
+    ]
+  },
+  "transformer.h.9.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 893.1614990234375,
+    "stable_rank": 115.87055206298828,
+    "spectral_norm": 17.444477081298828,
+    "frobenius_norm": 187.7779083251953,
+    "mp_bound": 12.937507738520768,
+    "n_above_mp": 18,
+    "n_total": 1024,
+    "signal_ratio": 0.017578125,
+    "alpha": 0.39390157964822387,
+    "alpha_r2": 0.8784313742375426,
+    "condition_number": 25.246517181396484,
+    "top_10_sv": [
+      17.444477081298828,
+      16.29485511779785,
+      15.905627250671387,
+      15.577847480773926,
+      15.356382369995117,
+      15.149186134338379,
+      14.64958381652832,
+      14.456648826599121,
+      14.237245559692383,
+      14.110804557800293
+    ]
+  },
+  "transformer.h.9.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 723.2728271484375,
+    "stable_rank": 13.839754104614258,
+    "spectral_norm": 26.413339614868164,
+    "frobenius_norm": 98.2624282836914,
+    "mp_bound": 3.006326913833618,
+    "n_above_mp": 307,
+    "n_total": 1024,
+    "signal_ratio": 0.2998046875,
+    "alpha": 0.5910035608611524,
+    "alpha_r2": 0.7777498557933287,
+    "condition_number": 84904.125,
+    "top_10_sv": [
+      26.413339614868164,
+      24.238210678100586,
+      13.290877342224121,
+      11.937247276306152,
+      9.426248550415039,
+      7.983802795410156,
+      7.864888668060303,
+      7.773645877838135,
+      7.429806232452393,
+      7.390601634979248
+    ]
+  },
+  "transformer.h.9.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 938.6810913085938,
+    "stable_rank": 35.63467025756836,
+    "spectral_norm": 35.846248626708984,
+    "frobenius_norm": 213.9833984375,
+    "mp_bound": 18.81469202041626,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.3128217781401333,
+    "alpha_r2": 0.9229840018865637,
+    "condition_number": 41.859554290771484,
+    "top_10_sv": [
+      35.846248626708984,
+      17.866443634033203,
+      17.141401290893555,
+      16.734554290771484,
+      16.55880355834961,
+      16.355411529541016,
+      15.886651039123535,
+      15.564071655273438,
+      15.157404899597168,
+      14.905134201049805
+    ]
+  },
+  "transformer.h.9.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 944.1468505859375,
+    "stable_rank": 77.85943603515625,
+    "spectral_norm": 21.045055389404297,
+    "frobenius_norm": 185.6973419189453,
+    "mp_bound": 16.68483567237854,
+    "n_above_mp": 4,
+    "n_total": 1024,
+    "signal_ratio": 0.00390625,
+    "alpha": 0.30334476439857244,
+    "alpha_r2": 0.9238160976321365,
+    "condition_number": 36.544837951660156,
+    "top_10_sv": [
+      21.045055389404297,
+      18.942150115966797,
+      18.65625762939453,
+      17.60894203186035,
+      14.828644752502441,
+      13.582180976867676,
+      13.063478469848633,
+      12.69278335571289,
+      12.27115535736084,
+      12.16966438293457
+    ]
+  },
+  "transformer.h.10.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 889.07373046875,
+    "stable_rank": 110.98599243164062,
+    "spectral_norm": 18.18061065673828,
+    "frobenius_norm": 191.53253173828125,
+    "mp_bound": 12.933648669067457,
+    "n_above_mp": 23,
+    "n_total": 1024,
+    "signal_ratio": 0.0224609375,
+    "alpha": 0.400968608999224,
+    "alpha_r2": 0.8759391614511947,
+    "condition_number": 22.077619552612305,
+    "top_10_sv": [
+      18.18061065673828,
+      17.37954330444336,
+      16.723297119140625,
+      16.557512283325195,
+      15.972862243652344,
+      15.706113815307617,
+      15.19486141204834,
+      14.927367210388184,
+      14.738473892211914,
+      14.608979225158691
+    ]
+  },
+  "transformer.h.10.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 712.9119262695312,
+    "stable_rank": 15.614805221557617,
+    "spectral_norm": 24.71717071533203,
+    "frobenius_norm": 97.67131805419922,
+    "mp_bound": 2.9057722091674805,
+    "n_above_mp": 303,
+    "n_total": 1024,
+    "signal_ratio": 0.2958984375,
+    "alpha": 0.6156612751434485,
+    "alpha_r2": 0.7966407254881575,
+    "condition_number": 26397.486328125,
+    "top_10_sv": [
+      24.71717071533203,
+      22.72795867919922,
+      11.450672149658203,
+      9.635822296142578,
+      9.428568840026855,
+      8.725939750671387,
+      8.629619598388672,
+      8.33855152130127,
+      8.053350448608398,
+      8.0328950881958
+    ]
+  },
+  "transformer.h.10.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 936.62158203125,
+    "stable_rank": 36.54176712036133,
+    "spectral_norm": 35.16852951049805,
+    "frobenius_norm": 212.593017578125,
+    "mp_bound": 18.578571796417236,
+    "n_above_mp": 2,
+    "n_total": 1024,
+    "signal_ratio": 0.001953125,
+    "alpha": 0.31837997506899607,
+    "alpha_r2": 0.9268627447940765,
+    "condition_number": 44.306236267089844,
+    "top_10_sv": [
+      35.16852951049805,
+      18.78311538696289,
+      17.198081970214844,
+      16.65778350830078,
+      16.474864959716797,
+      16.360870361328125,
+      15.851672172546387,
+      15.545867919921875,
+      15.253518104553223,
+      15.09296703338623
+    ]
+  },
+  "transformer.h.10.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 945.8907470703125,
+    "stable_rank": 78.20089721679688,
+    "spectral_norm": 21.427583694458008,
+    "frobenius_norm": 189.48684692382812,
+    "mp_bound": 17.147469520568848,
+    "n_above_mp": 3,
+    "n_total": 1024,
+    "signal_ratio": 0.0029296875,
+    "alpha": 0.30030447990075465,
+    "alpha_r2": 0.9246116625990694,
+    "condition_number": 53.54841232299805,
+    "top_10_sv": [
+      21.427583694458008,
+      20.07192611694336,
+      17.730005264282227,
+      15.819792747497559,
+      14.968161582946777,
+      13.369080543518066,
+      13.173439025878906,
+      13.08027458190918,
+      12.7400541305542,
+      12.428592681884766
+    ]
+  },
+  "transformer.h.11.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 901.9686279296875,
+    "stable_rank": 114.79285430908203,
+    "spectral_norm": 17.392602920532227,
+    "frobenius_norm": 186.34683227539062,
+    "mp_bound": 13.299353036480547,
+    "n_above_mp": 13,
+    "n_total": 1024,
+    "signal_ratio": 0.0126953125,
+    "alpha": 0.38050221069018897,
+    "alpha_r2": 0.8828344695891076,
+    "condition_number": 18.282512664794922,
+    "top_10_sv": [
+      17.392602920532227,
+      16.71450424194336,
+      15.829066276550293,
+      15.560785293579102,
+      15.137024879455566,
+      14.838809967041016,
+      14.30210018157959,
+      13.980681419372559,
+      13.925599098205566,
+      13.726691246032715
+    ]
+  },
+  "transformer.h.11.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 733.375,
+    "stable_rank": 42.236331939697266,
+    "spectral_norm": 15.064600944519043,
+    "frobenius_norm": 97.90406799316406,
+    "mp_bound": 3.1596755981445312,
+    "n_above_mp": 294,
+    "n_total": 1024,
+    "signal_ratio": 0.287109375,
+    "alpha": 0.5869072812022946,
+    "alpha_r2": 0.7743078879808162,
+    "condition_number": 10999.072265625,
+    "top_10_sv": [
+      15.064600944519043,
+      13.98853874206543,
+      11.859731674194336,
+      11.481588363647461,
+      8.733123779296875,
+      8.476235389709473,
+      8.284677505493164,
+      8.177391052246094,
+      7.880287170410156,
+      7.761397361755371
+    ]
+  },
+  "transformer.h.11.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 936.5640869140625,
+    "stable_rank": 36.39714050292969,
+    "spectral_norm": 35.354408264160156,
+    "frobenius_norm": 213.29330444335938,
+    "mp_bound": 18.617774963378906,
+    "n_above_mp": 2,
+    "n_total": 1024,
+    "signal_ratio": 0.001953125,
+    "alpha": 0.3180465298760983,
+    "alpha_r2": 0.9268863407776537,
+    "condition_number": 36.84086990356445,
+    "top_10_sv": [
+      35.354408264160156,
+      19.106882095336914,
+      17.004091262817383,
+      16.809144973754883,
+      16.472070693969727,
+      16.30332374572754,
+      15.732508659362793,
+      15.557268142700195,
+      15.403963088989258,
+      14.984481811523438
+    ]
+  },
+  "transformer.h.11.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 950.7989501953125,
+    "stable_rank": 109.07081604003906,
+    "spectral_norm": 18.535531997680664,
+    "frobenius_norm": 193.57949829101562,
+    "mp_bound": 17.80658483505249,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.2904225097637395,
+    "alpha_r2": 0.9180203572981726,
+    "condition_number": 31.789386749267578,
+    "top_10_sv": [
+      18.535531997680664,
+      17.2476749420166,
+      15.5623140335083,
+      15.329514503479004,
+      14.429676055908203,
+      13.832291603088379,
+      13.37085247039795,
+      13.245711326599121,
+      13.16356372833252,
+      13.034541130065918
+    ]
+  },
+  "transformer.h.12.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 900.2671508789062,
+    "stable_rank": 111.13557434082031,
+    "spectral_norm": 17.990798950195312,
+    "frobenius_norm": 189.66055297851562,
+    "mp_bound": 13.46277887415714,
+    "n_above_mp": 15,
+    "n_total": 1024,
+    "signal_ratio": 0.0146484375,
+    "alpha": 0.3849465327520483,
+    "alpha_r2": 0.8884130947195702,
+    "condition_number": 19.290571212768555,
+    "top_10_sv": [
+      17.990798950195312,
+      17.627941131591797,
+      16.81555938720703,
+      16.37408447265625,
+      15.790731430053711,
+      15.523019790649414,
+      14.99732494354248,
+      14.942817687988281,
+      14.576937675476074,
+      14.378122329711914
+    ]
+  },
+  "transformer.h.12.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 726.1136474609375,
+    "stable_rank": 67.91838836669922,
+    "spectral_norm": 11.730238914489746,
+    "frobenius_norm": 96.67196655273438,
+    "mp_bound": 2.9830989837646484,
+    "n_above_mp": 317,
+    "n_total": 1024,
+    "signal_ratio": 0.3095703125,
+    "alpha": 0.6079611377940939,
+    "alpha_r2": 0.7713425607206535,
+    "condition_number": 74660.875,
+    "top_10_sv": [
+      11.730238914489746,
+      11.002860069274902,
+      9.450277328491211,
+      9.339866638183594,
+      9.130253791809082,
+      8.70073413848877,
+      8.407955169677734,
+      8.365035057067871,
+      8.216530799865723,
+      8.057732582092285
+    ]
+  },
+  "transformer.h.12.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 937.1617431640625,
+    "stable_rank": 35.29595184326172,
+    "spectral_norm": 35.728485107421875,
+    "frobenius_norm": 212.26434326171875,
+    "mp_bound": 18.63219451904297,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.31249088591006335,
+    "alpha_r2": 0.9217926851101266,
+    "condition_number": 59.26630401611328,
+    "top_10_sv": [
+      35.728485107421875,
+      18.58656883239746,
+      16.78707504272461,
+      16.50104331970215,
+      16.170015335083008,
+      15.932202339172363,
+      15.695175170898438,
+      15.225218772888184,
+      14.931440353393555,
+      14.655704498291016
+    ]
+  },
+  "transformer.h.12.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 951.9144897460938,
+    "stable_rank": 125.68574523925781,
+    "spectral_norm": 17.590118408203125,
+    "frobenius_norm": 197.20220947265625,
+    "mp_bound": 18.2495698928833,
+    "n_above_mp": 0,
+    "n_total": 1024,
+    "signal_ratio": 0.0,
+    "alpha": 0.28659027982486085,
+    "alpha_r2": 0.9157493562146201,
+    "condition_number": 63.03939437866211,
+    "top_10_sv": [
+      17.590118408203125,
+      16.458555221557617,
+      15.0971040725708,
+      14.703747749328613,
+      14.40058708190918,
+      14.16800308227539,
+      13.817294120788574,
+      13.755056381225586,
+      13.447993278503418,
+      13.400959968566895
+    ]
+  },
+  "transformer.h.13.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 921.8739013671875,
+    "stable_rank": 122.69657897949219,
+    "spectral_norm": 17.3557071685791,
+    "frobenius_norm": 192.24655151367188,
+    "mp_bound": 14.840765654402492,
+    "n_above_mp": 5,
+    "n_total": 1024,
+    "signal_ratio": 0.0048828125,
+    "alpha": 0.34449500140802347,
+    "alpha_r2": 0.8890960763544924,
+    "condition_number": 12.950295448303223,
+    "top_10_sv": [
+      17.3557071685791,
+      16.527515411376953,
+      16.102203369140625,
+      15.533954620361328,
+      15.408553123474121,
+      14.486262321472168,
+      14.440102577209473,
+      14.294885635375977,
+      13.857573509216309,
+      13.616387367248535
+    ]
+  },
+  "transformer.h.13.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 745.8662109375,
+    "stable_rank": 69.44125366210938,
+    "spectral_norm": 11.393943786621094,
+    "frobenius_norm": 94.94735717773438,
+    "mp_bound": 3.1595468521118164,
+    "n_above_mp": 305,
+    "n_total": 1024,
+    "signal_ratio": 0.2978515625,
+    "alpha": 0.5692666342371,
+    "alpha_r2": 0.7468213049281058,
+    "condition_number": 11072.6982421875,
+    "top_10_sv": [
+      11.393943786621094,
+      10.033926010131836,
+      9.37811279296875,
+      8.2539701461792,
+      7.887532711029053,
+      7.69539213180542,
+      7.5529046058654785,
+      7.425933837890625,
+      7.274467468261719,
+      7.151112079620361
+    ]
+  },
+  "transformer.h.13.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 937.599365234375,
+    "stable_rank": 37.05495071411133,
+    "spectral_norm": 35.005916595458984,
+    "frobenius_norm": 213.0907440185547,
+    "mp_bound": 18.706077575683594,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.3122716285047514,
+    "alpha_r2": 0.9183302410504182,
+    "condition_number": 36.091087341308594,
+    "top_10_sv": [
+      35.005916595458984,
+      18.3486385345459,
+      16.548778533935547,
+      16.184619903564453,
+      15.928171157836914,
+      15.709515571594238,
+      15.531082153320312,
+      14.870841026306152,
+      14.759794235229492,
+      14.733899116516113
+    ]
+  },
+  "transformer.h.13.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 953.9815063476562,
+    "stable_rank": 93.02642059326172,
+    "spectral_norm": 21.067401885986328,
+    "frobenius_norm": 203.19552612304688,
+    "mp_bound": 18.922306537628174,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.2803336865969046,
+    "alpha_r2": 0.9048293558413308,
+    "condition_number": 39.748817443847656,
+    "top_10_sv": [
+      21.067401885986328,
+      17.05304718017578,
+      15.297992706298828,
+      14.769113540649414,
+      14.03395938873291,
+      13.835834503173828,
+      13.424732208251953,
+      13.155864715576172,
+      13.011979103088379,
+      12.918780326843262
+    ]
+  },
+  "transformer.h.14.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 927.090576171875,
+    "stable_rank": 125.47744750976562,
+    "spectral_norm": 16.747602462768555,
+    "frobenius_norm": 187.60113525390625,
+    "mp_bound": 14.835510292247621,
+    "n_above_mp": 4,
+    "n_total": 1024,
+    "signal_ratio": 0.00390625,
+    "alpha": 0.33493011773835984,
+    "alpha_r2": 0.893001234472107,
+    "condition_number": 14.26633358001709,
+    "top_10_sv": [
+      16.747602462768555,
+      15.885435104370117,
+      15.1227388381958,
+      14.847970008850098,
+      14.377248764038086,
+      14.23108959197998,
+      13.869467735290527,
+      13.69625186920166,
+      13.520434379577637,
+      13.335817337036133
+    ]
+  },
+  "transformer.h.14.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 771.0768432617188,
+    "stable_rank": 45.977813720703125,
+    "spectral_norm": 14.095232963562012,
+    "frobenius_norm": 95.57546997070312,
+    "mp_bound": 3.5943875312805176,
+    "n_above_mp": 260,
+    "n_total": 1024,
+    "signal_ratio": 0.25390625,
+    "alpha": 0.5172198054236867,
+    "alpha_r2": 0.7357139263927202,
+    "condition_number": 4705.93505859375,
+    "top_10_sv": [
+      14.095232963562012,
+      10.67358684539795,
+      9.503582000732422,
+      7.890437126159668,
+      7.194736003875732,
+      7.045958042144775,
+      7.003268718719482,
+      6.855443000793457,
+      6.832728862762451,
+      6.729796886444092
+    ]
+  },
+  "transformer.h.14.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 938.0935668945312,
+    "stable_rank": 32.70786666870117,
+    "spectral_norm": 37.25117492675781,
+    "frobenius_norm": 213.04241943359375,
+    "mp_bound": 18.850232362747192,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.3063623329883038,
+    "alpha_r2": 0.9141184802270717,
+    "condition_number": 55.86502456665039,
+    "top_10_sv": [
+      37.25117492675781,
+      17.538530349731445,
+      16.43047332763672,
+      15.928857803344727,
+      15.448025703430176,
+      15.416549682617188,
+      15.191793441772461,
+      14.8709135055542,
+      14.701964378356934,
+      14.292315483093262
+    ]
+  },
+  "transformer.h.14.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 954.6704711914062,
+    "stable_rank": 85.12628173828125,
+    "spectral_norm": 23.248565673828125,
+    "frobenius_norm": 214.50035095214844,
+    "mp_bound": 19.985750913619995,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.2766327750388291,
+    "alpha_r2": 0.9013307804087409,
+    "condition_number": 80.55670166015625,
+    "top_10_sv": [
+      23.248565673828125,
+      18.41067886352539,
+      15.474272727966309,
+      14.885811805725098,
+      14.50477409362793,
+      14.26229476928711,
+      14.121097564697266,
+      13.90327262878418,
+      13.509455680847168,
+      13.493183135986328
+    ]
+  },
+  "transformer.h.15.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 927.6941528320312,
+    "stable_rank": 123.29837799072266,
+    "spectral_norm": 16.74991798400879,
+    "frobenius_norm": 185.99075317382812,
+    "mp_bound": 14.70581637928394,
+    "n_above_mp": 3,
+    "n_total": 1024,
+    "signal_ratio": 0.0029296875,
+    "alpha": 0.33338608542644904,
+    "alpha_r2": 0.889507619370194,
+    "condition_number": 13.823485374450684,
+    "top_10_sv": [
+      16.74991798400879,
+      15.065845489501953,
+      14.728111267089844,
+      14.221285820007324,
+      14.000044822692871,
+      13.83415412902832,
+      13.696669578552246,
+      13.493361473083496,
+      13.296980857849121,
+      13.158197402954102
+    ]
+  },
+  "transformer.h.15.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 734.9755859375,
+    "stable_rank": 39.57041549682617,
+    "spectral_norm": 15.907938957214355,
+    "frobenius_norm": 100.06892395019531,
+    "mp_bound": 3.168710231781006,
+    "n_above_mp": 315,
+    "n_total": 1024,
+    "signal_ratio": 0.3076171875,
+    "alpha": 0.5890122729886013,
+    "alpha_r2": 0.7620616339931927,
+    "condition_number": 118929.2421875,
+    "top_10_sv": [
+      15.907938957214355,
+      9.037424087524414,
+      8.875422477722168,
+      8.679743766784668,
+      8.60929012298584,
+      8.428465843200684,
+      8.394245147705078,
+      8.310733795166016,
+      8.301813125610352,
+      8.17810344696045
+    ]
+  },
+  "transformer.h.15.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 938.0027465820312,
+    "stable_rank": 33.95222854614258,
+    "spectral_norm": 36.603248596191406,
+    "frobenius_norm": 213.28179931640625,
+    "mp_bound": 18.90978169441223,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.30609205088241465,
+    "alpha_r2": 0.9144551505747684,
+    "condition_number": 63.18374252319336,
+    "top_10_sv": [
+      36.603248596191406,
+      17.053295135498047,
+      16.544368743896484,
+      16.148056030273438,
+      16.080337524414062,
+      15.509836196899414,
+      15.409363746643066,
+      15.122523307800293,
+      14.887523651123047,
+      14.568466186523438
+    ]
+  },
+  "transformer.h.15.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 954.9154663085938,
+    "stable_rank": 67.59476470947266,
+    "spectral_norm": 27.46138572692871,
+    "frobenius_norm": 225.77662658691406,
+    "mp_bound": 20.990967750549316,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.2748501163371319,
+    "alpha_r2": 0.8922496782749698,
+    "condition_number": 75.69937896728516,
+    "top_10_sv": [
+      27.46138572692871,
+      20.168291091918945,
+      16.48451805114746,
+      14.961456298828125,
+      14.724319458007812,
+      14.283745765686035,
+      14.013591766357422,
+      13.970895767211914,
+      13.828929901123047,
+      13.712251663208008
+    ]
+  },
+  "transformer.h.16.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 946.042724609375,
+    "stable_rank": 144.74566650390625,
+    "spectral_norm": 15.667328834533691,
+    "frobenius_norm": 188.4940948486328,
+    "mp_bound": 15.990341963357688,
+    "n_above_mp": 0,
+    "n_total": 1024,
+    "signal_ratio": 0.0,
+    "alpha": 0.29235290161143895,
+    "alpha_r2": 0.8708847197371617,
+    "condition_number": 10.925378799438477,
+    "top_10_sv": [
+      15.667328834533691,
+      13.777777671813965,
+      13.580466270446777,
+      13.480743408203125,
+      13.149480819702148,
+      12.728473663330078,
+      12.464872360229492,
+      12.249978065490723,
+      12.104480743408203,
+      12.058478355407715
+    ]
+  },
+  "transformer.h.16.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 771.8905639648438,
+    "stable_rank": 34.916778564453125,
+    "spectral_norm": 17.508285522460938,
+    "frobenius_norm": 103.45719909667969,
+    "mp_bound": 3.9519896507263184,
+    "n_above_mp": 251,
+    "n_total": 1024,
+    "signal_ratio": 0.2451171875,
+    "alpha": 0.5150372729977702,
+    "alpha_r2": 0.7343136754490875,
+    "condition_number": 7742.5263671875,
+    "top_10_sv": [
+      17.508285522460938,
+      8.428668022155762,
+      8.318235397338867,
+      7.7397379875183105,
+      7.588472843170166,
+      7.562111854553223,
+      7.487995624542236,
+      7.291107177734375,
+      7.21236515045166,
+      7.17236852645874
+    ]
+  },
+  "transformer.h.16.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 940.1898193359375,
+    "stable_rank": 35.53369140625,
+    "spectral_norm": 35.824527740478516,
+    "frobenius_norm": 213.55052185058594,
+    "mp_bound": 19.066128730773926,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.29995952857301905,
+    "alpha_r2": 0.9104848364911763,
+    "condition_number": 64.50019836425781,
+    "top_10_sv": [
+      35.824527740478516,
+      16.836957931518555,
+      16.33154296875,
+      15.77452564239502,
+      15.625938415527344,
+      15.192853927612305,
+      14.959126472473145,
+      14.744555473327637,
+      14.429903030395508,
+      14.108305931091309
+    ]
+  },
+  "transformer.h.16.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 953.3158569335938,
+    "stable_rank": 65.17062377929688,
+    "spectral_norm": 29.556535720825195,
+    "frobenius_norm": 238.60496520996094,
+    "mp_bound": 22.07623028755188,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.27669229111220206,
+    "alpha_r2": 0.8878361607414883,
+    "condition_number": 59.69606018066406,
+    "top_10_sv": [
+      29.556535720825195,
+      20.332595825195312,
+      17.3956356048584,
+      15.779086112976074,
+      15.581547737121582,
+      15.288317680358887,
+      15.058049201965332,
+      14.879526138305664,
+      14.627667427062988,
+      14.318242073059082
+    ]
+  },
+  "transformer.h.17.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 947.1865234375,
+    "stable_rank": 142.42913818359375,
+    "spectral_norm": 15.703680992126465,
+    "frobenius_norm": 187.4134979248047,
+    "mp_bound": 15.91192969963621,
+    "n_above_mp": 0,
+    "n_total": 1024,
+    "signal_ratio": 0.0,
+    "alpha": 0.28839905133263977,
+    "alpha_r2": 0.8639671147195912,
+    "condition_number": 9.761466026306152,
+    "top_10_sv": [
+      15.703680992126465,
+      14.009387016296387,
+      13.541427612304688,
+      13.323938369750977,
+      12.849181175231934,
+      12.739950180053711,
+      12.229618072509766,
+      12.002062797546387,
+      11.895586013793945,
+      11.779114723205566
+    ]
+  },
+  "transformer.h.17.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 761.0531005859375,
+    "stable_rank": 24.522497177124023,
+    "spectral_norm": 21.281356811523438,
+    "frobenius_norm": 105.38569641113281,
+    "mp_bound": 3.7680649757385254,
+    "n_above_mp": 283,
+    "n_total": 1024,
+    "signal_ratio": 0.2763671875,
+    "alpha": 0.5307534843077817,
+    "alpha_r2": 0.7242129172164211,
+    "condition_number": 3630.01904296875,
+    "top_10_sv": [
+      21.281356811523438,
+      8.807740211486816,
+      8.406106948852539,
+      8.249408721923828,
+      7.748202323913574,
+      7.372308254241943,
+      7.20876407623291,
+      7.15388298034668,
+      7.152019500732422,
+      7.0919060707092285
+    ]
+  },
+  "transformer.h.17.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 941.4163208007812,
+    "stable_rank": 32.30669403076172,
+    "spectral_norm": 37.66846466064453,
+    "frobenius_norm": 214.10369873046875,
+    "mp_bound": 19.23757553100586,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.29567148822186234,
+    "alpha_r2": 0.9092601420648001,
+    "condition_number": 106.81797790527344,
+    "top_10_sv": [
+      37.66846466064453,
+      16.419097900390625,
+      15.927478790283203,
+      15.601706504821777,
+      15.464677810668945,
+      15.215024948120117,
+      14.866231918334961,
+      14.730674743652344,
+      14.174066543579102,
+      14.107979774475098
+    ]
+  },
+  "transformer.h.17.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 955.3389892578125,
+    "stable_rank": 56.71522521972656,
+    "spectral_norm": 32.85487747192383,
+    "frobenius_norm": 247.42848205566406,
+    "mp_bound": 23.075778007507324,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.26997116277589567,
+    "alpha_r2": 0.8814747074927219,
+    "condition_number": 65.88758850097656,
+    "top_10_sv": [
+      32.85487747192383,
+      20.14518165588379,
+      17.330669403076172,
+      16.186241149902344,
+      15.963332176208496,
+      15.739975929260254,
+      15.285326957702637,
+      14.80821418762207,
+      14.750692367553711,
+      14.634135246276855
+    ]
+  },
+  "transformer.h.18.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 946.826171875,
+    "stable_rank": 142.5345458984375,
+    "spectral_norm": 15.78872013092041,
+    "frobenius_norm": 188.49810791015625,
+    "mp_bound": 15.980405578724305,
+    "n_above_mp": 0,
+    "n_total": 1024,
+    "signal_ratio": 0.0,
+    "alpha": 0.29059627057789356,
+    "alpha_r2": 0.8667678175025767,
+    "condition_number": 8.09025764465332,
+    "top_10_sv": [
+      15.78872013092041,
+      13.945246696472168,
+      13.766387939453125,
+      13.405774116516113,
+      13.049812316894531,
+      12.528767585754395,
+      12.479240417480469,
+      12.379585266113281,
+      12.237228393554688,
+      12.036273956298828
+    ]
+  },
+  "transformer.h.18.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 786.9091796875,
+    "stable_rank": 25.3469181060791,
+    "spectral_norm": 21.475971221923828,
+    "frobenius_norm": 108.12232971191406,
+    "mp_bound": 4.413455486297607,
+    "n_above_mp": 222,
+    "n_total": 1024,
+    "signal_ratio": 0.216796875,
+    "alpha": 0.478483490126483,
+    "alpha_r2": 0.7161237347700887,
+    "condition_number": 69973.6328125,
+    "top_10_sv": [
+      21.475971221923828,
+      8.631346702575684,
+      8.211710929870605,
+      8.147095680236816,
+      7.6618499755859375,
+      7.513932228088379,
+      7.377252101898193,
+      7.279123783111572,
+      7.242250919342041,
+      7.182393550872803
+    ]
+  },
+  "transformer.h.18.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 944.5552978515625,
+    "stable_rank": 33.118221282958984,
+    "spectral_norm": 37.25223159790039,
+    "frobenius_norm": 214.38075256347656,
+    "mp_bound": 19.486257076263428,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.289183381623581,
+    "alpha_r2": 0.9070325638595973,
+    "condition_number": 98.16948699951172,
+    "top_10_sv": [
+      37.25223159790039,
+      15.790482521057129,
+      15.473872184753418,
+      15.28934097290039,
+      15.085716247558594,
+      14.840494155883789,
+      14.610716819763184,
+      14.552131652832031,
+      14.460594177246094,
+      14.226823806762695
+    ]
+  },
+  "transformer.h.18.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 955.5623168945312,
+    "stable_rank": 62.169673919677734,
+    "spectral_norm": 32.66622543334961,
+    "frobenius_norm": 257.5658264160156,
+    "mp_bound": 24.027284145355225,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.2709203055980388,
+    "alpha_r2": 0.8864060504096509,
+    "condition_number": 42.17201232910156,
+    "top_10_sv": [
+      32.66622543334961,
+      21.241456985473633,
+      18.316198348999023,
+      18.020597457885742,
+      17.09003257751465,
+      16.6181583404541,
+      16.53766632080078,
+      16.19798469543457,
+      15.890321731567383,
+      15.60676383972168
+    ]
+  },
+  "transformer.h.19.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 942.326171875,
+    "stable_rank": 155.50115966796875,
+    "spectral_norm": 14.992883682250977,
+    "frobenius_norm": 186.96141052246094,
+    "mp_bound": 15.57602523915173,
+    "n_above_mp": 0,
+    "n_total": 1024,
+    "signal_ratio": 0.0,
+    "alpha": 0.2982009204884547,
+    "alpha_r2": 0.8572427405933445,
+    "condition_number": 7.912736892700195,
+    "top_10_sv": [
+      14.992883682250977,
+      14.037845611572266,
+      13.398187637329102,
+      13.230586051940918,
+      12.737166404724121,
+      12.300458908081055,
+      12.292224884033203,
+      12.19207763671875,
+      12.00888729095459,
+      11.759111404418945
+    ]
+  },
+  "transformer.h.19.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 807.1912231445312,
+    "stable_rank": 17.76304054260254,
+    "spectral_norm": 27.394882202148438,
+    "frobenius_norm": 115.45907592773438,
+    "mp_bound": 5.350113868713379,
+    "n_above_mp": 135,
+    "n_total": 1024,
+    "signal_ratio": 0.1318359375,
+    "alpha": 0.4361821075722079,
+    "alpha_r2": 0.7264706510936031,
+    "condition_number": 15827.857421875,
+    "top_10_sv": [
+      27.394882202148438,
+      9.131587028503418,
+      8.159581184387207,
+      7.967867374420166,
+      7.304231643676758,
+      7.250730037689209,
+      7.2139668464660645,
+      7.170718669891357,
+      7.162339687347412,
+      7.095859050750732
+    ]
+  },
+  "transformer.h.19.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 946.293212890625,
+    "stable_rank": 33.909671783447266,
+    "spectral_norm": 36.853206634521484,
+    "frobenius_norm": 214.6036376953125,
+    "mp_bound": 19.577980041503906,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.2855408853304053,
+    "alpha_r2": 0.9022405306041533,
+    "condition_number": 91.69256591796875,
+    "top_10_sv": [
+      36.853206634521484,
+      15.805397033691406,
+      15.353994369506836,
+      15.227054595947266,
+      15.066706657409668,
+      14.814555168151855,
+      14.390864372253418,
+      14.297786712646484,
+      14.181514739990234,
+      13.928579330444336
+    ]
+  },
+  "transformer.h.19.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 956.4804077148438,
+    "stable_rank": 52.073970794677734,
+    "spectral_norm": 37.2791862487793,
+    "frobenius_norm": 269.0151672363281,
+    "mp_bound": 25.231703281402588,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.26739284885206765,
+    "alpha_r2": 0.8835308501225437,
+    "condition_number": 45.95317077636719,
+    "top_10_sv": [
+      37.2791862487793,
+      20.44887351989746,
+      19.732032775878906,
+      18.161226272583008,
+      17.561620712280273,
+      17.188261032104492,
+      17.060258865356445,
+      16.776437759399414,
+      16.5764217376709,
+      16.08405303955078
+    ]
+  },
+  "transformer.h.20.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 936.0965576171875,
+    "stable_rank": 156.9998779296875,
+    "spectral_norm": 14.947098731994629,
+    "frobenius_norm": 187.28652954101562,
+    "mp_bound": 15.20929343045714,
+    "n_above_mp": 0,
+    "n_total": 1024,
+    "signal_ratio": 0.0,
+    "alpha": 0.3120911362337675,
+    "alpha_r2": 0.858786145933246,
+    "condition_number": 8.875840187072754,
+    "top_10_sv": [
+      14.947098731994629,
+      14.385692596435547,
+      13.843692779541016,
+      13.672863960266113,
+      13.224756240844727,
+      12.674160957336426,
+      12.55836296081543,
+      12.521924018859863,
+      12.441327095031738,
+      11.893940925598145
+    ]
+  },
+  "transformer.h.20.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 801.7127685546875,
+    "stable_rank": 16.856487274169922,
+    "spectral_norm": 28.67116355895996,
+    "frobenius_norm": 117.71420288085938,
+    "mp_bound": 5.2284698486328125,
+    "n_above_mp": 163,
+    "n_total": 1024,
+    "signal_ratio": 0.1591796875,
+    "alpha": 0.444035142929919,
+    "alpha_r2": 0.7121527583409499,
+    "condition_number": 31572.4609375,
+    "top_10_sv": [
+      28.67116355895996,
+      9.656560897827148,
+      8.204039573669434,
+      7.896355152130127,
+      7.5075554847717285,
+      7.466111660003662,
+      7.3787007331848145,
+      7.314346790313721,
+      7.256669044494629,
+      7.230027675628662
+    ]
+  },
+  "transformer.h.20.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 948.0650024414062,
+    "stable_rank": 35.34494400024414,
+    "spectral_norm": 36.2116813659668,
+    "frobenius_norm": 215.2843017578125,
+    "mp_bound": 19.70071792602539,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.2835115515690549,
+    "alpha_r2": 0.8999757829277988,
+    "condition_number": 68.69733428955078,
+    "top_10_sv": [
+      36.2116813659668,
+      15.233745574951172,
+      15.040852546691895,
+      15.021078109741211,
+      14.903677940368652,
+      14.672463417053223,
+      14.484726905822754,
+      14.234298706054688,
+      13.947386741638184,
+      13.761749267578125
+    ]
+  },
+  "transformer.h.20.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 957.45361328125,
+    "stable_rank": 53.200496673583984,
+    "spectral_norm": 38.38079071044922,
+    "frobenius_norm": 279.94439697265625,
+    "mp_bound": 26.34588861465454,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.26655693080921455,
+    "alpha_r2": 0.8858868317060345,
+    "condition_number": 45.9372444152832,
+    "top_10_sv": [
+      38.38079071044922,
+      20.71181869506836,
+      19.3667049407959,
+      19.097551345825195,
+      18.877281188964844,
+      18.161039352416992,
+      17.817678451538086,
+      17.619935989379883,
+      17.085142135620117,
+      17.002307891845703
+    ]
+  },
+  "transformer.h.21.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 941.8446044921875,
+    "stable_rank": 150.69029235839844,
+    "spectral_norm": 15.391748428344727,
+    "frobenius_norm": 188.94290161132812,
+    "mp_bound": 15.802790776950205,
+    "n_above_mp": 0,
+    "n_total": 1024,
+    "signal_ratio": 0.0,
+    "alpha": 0.3000971543853719,
+    "alpha_r2": 0.8719087171149849,
+    "condition_number": 11.076602935791016,
+    "top_10_sv": [
+      15.391748428344727,
+      14.966634750366211,
+      14.548294067382812,
+      13.83862018585205,
+      13.741484642028809,
+      13.272273063659668,
+      13.161110877990723,
+      12.848255157470703,
+      12.513276100158691,
+      12.309864044189453
+    ]
+  },
+  "transformer.h.21.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 802.9078979492188,
+    "stable_rank": 10.218631744384766,
+    "spectral_norm": 38.61031723022461,
+    "frobenius_norm": 123.42403411865234,
+    "mp_bound": 5.600226402282715,
+    "n_above_mp": 132,
+    "n_total": 1024,
+    "signal_ratio": 0.12890625,
+    "alpha": 0.43585709299085673,
+    "alpha_r2": 0.7264214720046629,
+    "condition_number": 16619.98828125,
+    "top_10_sv": [
+      38.61031723022461,
+      12.657886505126953,
+      9.889753341674805,
+      9.055880546569824,
+      8.717723846435547,
+      7.8169941902160645,
+      7.405618190765381,
+      7.38328742980957,
+      7.281287670135498,
+      7.214484691619873
+    ]
+  },
+  "transformer.h.21.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 947.9262084960938,
+    "stable_rank": 36.02553176879883,
+    "spectral_norm": 35.89358139038086,
+    "frobenius_norm": 215.43785095214844,
+    "mp_bound": 19.718321800231934,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.2838725389249319,
+    "alpha_r2": 0.8980818942031933,
+    "condition_number": 72.88961029052734,
+    "top_10_sv": [
+      35.89358139038086,
+      15.480804443359375,
+      14.935028076171875,
+      14.85132122039795,
+      14.700712203979492,
+      14.630121231079102,
+      14.243003845214844,
+      14.099127769470215,
+      13.748274803161621,
+      13.597418785095215
+    ]
+  },
+  "transformer.h.21.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 957.150146484375,
+    "stable_rank": 40.031089782714844,
+    "spectral_norm": 45.864524841308594,
+    "frobenius_norm": 290.1854248046875,
+    "mp_bound": 27.241085529327393,
+    "n_above_mp": 2,
+    "n_total": 1024,
+    "signal_ratio": 0.001953125,
+    "alpha": 0.26491965004813967,
+    "alpha_r2": 0.8792307905237831,
+    "condition_number": 55.03404998779297,
+    "top_10_sv": [
+      45.864524841308594,
+      28.97075080871582,
+      19.798044204711914,
+      18.97209358215332,
+      18.70978355407715,
+      18.018537521362305,
+      17.47182273864746,
+      17.274417877197266,
+      16.987682342529297,
+      16.63946533203125
+    ]
+  },
+  "transformer.h.22.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 910.2899780273438,
+    "stable_rank": 125.6297836303711,
+    "spectral_norm": 16.882688522338867,
+    "frobenius_norm": 189.22909545898438,
+    "mp_bound": 14.07356805741784,
+    "n_above_mp": 5,
+    "n_total": 1024,
+    "signal_ratio": 0.0048828125,
+    "alpha": 0.3631109400306433,
+    "alpha_r2": 0.8785417413420795,
+    "condition_number": 18.38054656982422,
+    "top_10_sv": [
+      16.882688522338867,
+      15.70670223236084,
+      15.6824312210083,
+      14.943181037902832,
+      14.69243335723877,
+      13.932169914245605,
+      13.615700721740723,
+      13.460670471191406,
+      13.248589515686035,
+      12.95559024810791
+    ]
+  },
+  "transformer.h.22.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 791.9282836914062,
+    "stable_rank": 7.863592624664307,
+    "spectral_norm": 47.40917205810547,
+    "frobenius_norm": 132.9452667236328,
+    "mp_bound": 5.678843975067139,
+    "n_above_mp": 149,
+    "n_total": 1024,
+    "signal_ratio": 0.1455078125,
+    "alpha": 0.4420612606508839,
+    "alpha_r2": 0.7218298086729467,
+    "condition_number": 11620.599609375,
+    "top_10_sv": [
+      47.40917205810547,
+      18.899751663208008,
+      13.706535339355469,
+      12.33392333984375,
+      10.662578582763672,
+      8.17455768585205,
+      7.832640171051025,
+      7.784000396728516,
+      7.671928882598877,
+      7.559737682342529
+    ]
+  },
+  "transformer.h.22.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 947.9542846679688,
+    "stable_rank": 35.02265548706055,
+    "spectral_norm": 36.702762603759766,
+    "frobenius_norm": 217.2067413330078,
+    "mp_bound": 19.8194797039032,
+    "n_above_mp": 1,
+    "n_total": 1024,
+    "signal_ratio": 0.0009765625,
+    "alpha": 0.28619079431362643,
+    "alpha_r2": 0.9002755587642126,
+    "condition_number": 57.37492752075195,
+    "top_10_sv": [
+      36.702762603759766,
+      15.816225051879883,
+      15.100847244262695,
+      14.771247863769531,
+      14.534582138061523,
+      14.363577842712402,
+      14.114562034606934,
+      14.111546516418457,
+      13.981919288635254,
+      13.779252052307129
+    ]
+  },
+  "transformer.h.22.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 947.5176391601562,
+    "stable_rank": 17.500995635986328,
+    "spectral_norm": 73.44800567626953,
+    "frobenius_norm": 307.2637939453125,
+    "mp_bound": 27.96702003479004,
+    "n_above_mp": 3,
+    "n_total": 1024,
+    "signal_ratio": 0.0029296875,
+    "alpha": 0.27600893237986174,
+    "alpha_r2": 0.8907517199504488,
+    "condition_number": 77.1149673461914,
+    "top_10_sv": [
+      73.44800567626953,
+      46.17403030395508,
+      41.175079345703125,
+      19.932249069213867,
+      19.340335845947266,
+      18.94622039794922,
+      18.65909767150879,
+      17.994863510131836,
+      17.704139709472656,
+      17.6275577545166
+    ]
+  },
+  "transformer.h.23.attn.c_attn.weight": {
+    "shape": [
+      1024,
+      3072
+    ],
+    "effective_rank": 920.36767578125,
+    "stable_rank": 83.51404571533203,
+    "spectral_norm": 20.953210830688477,
+    "frobenius_norm": 191.48306274414062,
+    "mp_bound": 14.838865915472999,
+    "n_above_mp": 6,
+    "n_total": 1024,
+    "signal_ratio": 0.005859375,
+    "alpha": 0.347956924436965,
+    "alpha_r2": 0.9030461026469062,
+    "condition_number": 30.88874626159668,
+    "top_10_sv": [
+      20.953210830688477,
+      18.181076049804688,
+      16.9401798248291,
+      16.4212703704834,
+      16.156402587890625,
+      15.156576156616211,
+      14.803506851196289,
+      14.501842498779297,
+      14.308381080627441,
+      14.216387748718262
+    ]
+  },
+  "transformer.h.23.attn.c_proj.weight": {
+    "shape": [
+      1024,
+      1024
+    ],
+    "effective_rank": 725.8294067382812,
+    "stable_rank": 7.097742080688477,
+    "spectral_norm": 54.51199722290039,
+    "frobenius_norm": 145.22862243652344,
+    "mp_bound": 4.808560848236084,
+    "n_above_mp": 205,
+    "n_total": 1024,
+    "signal_ratio": 0.2001953125,
+    "alpha": 0.5165891829463887,
+    "alpha_r2": 0.7805439618628243,
+    "condition_number": 81928.5625,
+    "top_10_sv": [
+      54.51199722290039,
+      51.59781265258789,
+      31.993318557739258,
+      28.781885147094727,
+      20.638469696044922,
+      19.69414520263672,
+      17.257286071777344,
+      11.122438430786133,
+      9.82822036743164,
+      8.54098129272461
+    ]
+  },
+  "transformer.h.23.mlp.c_fc.weight": {
+    "shape": [
+      1024,
+      4096
+    ],
+    "effective_rank": 949.3385009765625,
+    "stable_rank": 32.35031509399414,
+    "spectral_norm": 39.07307052612305,
+    "frobenius_norm": 222.2372283935547,
+    "mp_bound": 20.35976243019104,
+    "n_above_mp": 2,
+    "n_total": 1024,
+    "signal_ratio": 0.001953125,
+    "alpha": 0.2880342497839811,
+    "alpha_r2": 0.9166713464789396,
+    "condition_number": 156.26080322265625,
+    "top_10_sv": [
+      39.07307052612305,
+      21.5715389251709,
+      18.421354293823242,
+      18.168561935424805,
+      16.698549270629883,
+      16.427534103393555,
+      15.287932395935059,
+      14.903624534606934,
+      14.39078140258789,
+      14.156229019165039
+    ]
+  },
+  "transformer.h.23.mlp.c_proj.weight": {
+    "shape": [
+      4096,
+      1024
+    ],
+    "effective_rank": 952.0796508789062,
+    "stable_rank": 23.627531051635742,
+    "spectral_norm": 65.28954315185547,
+    "frobenius_norm": 317.3604431152344,
+    "mp_bound": 29.342853784561157,
+    "n_above_mp": 3,
+    "n_total": 1024,
+    "signal_ratio": 0.0029296875,
+    "alpha": 0.2737939337158801,
+    "alpha_r2": 0.9043915028204278,
+    "condition_number": 46.49357223510742,
+    "top_10_sv": [
+      65.28954315185547,
+      45.984188079833984,
+      36.61665725708008,
+      27.076080322265625,
+      23.387649536132812,
+      21.129486083984375,
+      20.738719940185547,
+      20.499759674072266,
+      20.096046447753906,
+      19.36311912536621
+    ]
+  }
+}