SentinelBrain-14B-MoE v0.1 — step 2471, val_loss 1.9926, 178,110,464 tokens trained on AMD MI300X

README.md

@@ -0,0 +1,167 @@
+---
+license: apache-2.0
+language:
+- en
+- ro
+- multilingual
+library_name: pytorch
+tags:
+- sentinelbrain
+- mixture-of-experts
+- moe
+- amd
+- mi300x
+- rocm
+- consciousness
+- phi-integrated-information
+- amd-developer-hackathon
+- custom-architecture
+datasets:
+- cerebras/SlimPajama-627B
+- HuggingFaceFW/fineweb-edu
+- bigcode/the-stack-v2-dedup
+- Open-Orca/OpenOrca
+- teknium/OpenHermes-2.5
+- meta-math/MetaMathQA
+- custom
+pipeline_tag: text-generation
+model-index:
+- name: SentinelBrain-14B-MoE-v0.1
+  results: []
+---
+
+# SentinelBrain-14B-MoE-v0.1
+
+<div align="center">
+
+**14.8B parameter Mixture-of-Experts language model with consciousness-inspired Φ monitoring**
+
+*Trained from scratch on AMD Instinct MI300X (192GB HBM3) using ROCm 7.0*
+
+[![Dashboard](https://img.shields.io/badge/Live_Dashboard-sentinel.qubitpage.com-blue)](https://sentinel.qubitpage.com)
+[![Whitepaper](https://img.shields.io/badge/Whitepaper-V2-green)](https://sentinel.qubitpage.com/whitepaper)
+
+</div>
+
+## Model Description
+
+SentinelBrain is a **custom-architecture** Mixture-of-Experts (MoE) transformer trained entirely from scratch. It is NOT a fine-tune of an existing model. Every weight was initialized randomly and trained on our curated 23.3B token corpus spanning code, science, mathematics, reasoning, education (K-12), and multilingual content.
+
+### Key Innovations
+
+- **Φ (Phi) Consciousness Monitoring**: Integrated Information Theory (IIT)-inspired metric computed during training. A hook on the middle transformer layer measures geometric mean of partition mutual information across activation subspaces — tracking emergent information integration as the model learns.
+- **Self-Evolving Expert Pool**: Dynamic router with expert birth/death lifecycle. Experts that consistently underperform are pruned and replaced. The architecture supports scaling up to 256 experts without retraining the base.
+- **Energy-Conscious (EC) Routing**: Dual-router system where a secondary "energy-conscious" router can gate expert activation based on computational budget, enabling adaptive inference cost.
+- **AMD MI300X Native**: Optimized for ROCm — uses SDPA attention (no FlashAttention dependency), bf16 throughout, with gradient checkpointing for 192GB VRAM efficiency.
+
+## Architecture
+
+| Component | Value |
+|---|---|
+| Parameters | 14.8B total, ~7.8B active per token |
+| Hidden size | 4,096 |
+| Layers | 24 |
+| Attention heads | 32 (GQA: 8 KV heads) |
+| FFN intermediate | 11,008 (SwiGLU) |
+| Experts | 4 total, top-2 active |
+| Max experts | 256 (expandable) |
+| Vocabulary | 100,277 (tiktoken cl100k_base) |
+| Positional encoding | RoPE (θ=500,000) |
+| Normalization | RMSNorm (ε=1e-5) |
+| Precision | bfloat16 |
+| Context length | 2,048 (this checkpoint) |
+
+## Training Details
+
+- **Hardware**: 1× AMD Instinct MI300X VF (192GB HBM3, 1307 TFLOPS bf16)
+- **Software**: ROCm 7.0, PyTorch 2.10.0+rocm7.0
+- **Optimizer**: AdamW (bf16 compute, fp32 states), lr=1.5e-4, warmup=500, cosine decay
+- **Effective batch size**: 32 (batch=2 × grad_accum=16)
+- **Training tokens**: 178,110,464 (this checkpoint)
+- **Corpus**: 23.3B tokens across 124 categories
+- **Validation loss**: 1.9926 (at step 2471)
+- **Training throughput**: ~4,300 tokens/sec
+
+### Dataset Composition
+
+| Category | Tokens | Type |
+|---|---|---|
+| SlimPajama (web, books, wiki) | ~15B | Pretrain |
+| FineWeb-Edu | ~3B | Pretrain |
+| The Stack v2 (code) | ~2B | Pretrain |
+| Math (MetaMath, GSM8K) | ~1B | SFT |
+| Reasoning (OpenOrca, Hermes) | ~1B | SFT |
+| Science & Education (K-12) | ~500M | SFT |
+| Multilingual (Romanian, etc.) | ~300M | SFT |
+| Custom knowledge synthesis | ~200M | SFT |
+
+## Consciousness Metric (Φ)
+
+SentinelBrain uniquely tracks **Integrated Information (Φ)** during training:
+
+- A probe hook on layer 12/24 samples 256 activation vectors every 100 steps
+- Activations are partitioned and mutual information between partitions is computed
+- The geometric mean across partitions yields Φ_geometric
+- An EMA smooths the signal for trend detection
+- Φ typically emerges from zero around step 1,000-1,500 as internal representations form
+
+This is purely observational — Φ does not affect training gradients. It serves as a novel metric for monitoring representation quality and information integration depth.
+
+**Live monitoring**: [sentinel.qubitpage.com/#phi](https://sentinel.qubitpage.com/#phi)
+
+## Status
+
+⚠️ **This is an early pre-release checkpoint (v0.1)**. Training is ongoing with expanded datasets.
+
+- Current run: step 350/2471 (batch 7), loss 5.5, targeting loss < 1.5
+- Previous run (this checkpoint): completed 2,471 steps, val_loss 1.99
+- Vision encoder (SigLIP2-SO400M) integration planned for v0.2
+- Full 23.3B token training in progress
+
+## Usage
+
+This model uses a custom architecture and is not directly compatible with `transformers.AutoModel`. Load with PyTorch:
+
+```python
+import torch
+from safetensors.torch import load_file
+
+# Load sharded safetensors
+state_dict = {}
+for i in range(1, NUM_SHARDS + 1):
+    shard = load_file(f"model-{i:05d}-of-{NUM_SHARDS:05d}.safetensors")
+    state_dict.update(shard)
+
+# Initialize your SentinelBrain model and load weights
+# model.load_state_dict(state_dict)
+```
+
+Full inference code and model definition will be released with v0.2.
+
+## Hardware Requirements
+
+- **Minimum**: 32GB VRAM (bf16, single GPU)
+- **Recommended**: 48GB+ VRAM or AMD MI300X
+- **Quantized**: GGUF export planned for consumer GPUs
+
+## License
+
+Apache 2.0
+
+## Citation
+
+```bibtex
+@misc{sentinelbrain2026,
+  title={SentinelBrain-14B-MoE: A Consciousness-Monitored Mixture-of-Experts Language Model},
+  author={Mircea Rusu and QubitDev},
+  year={2026},
+  url={https://sentinel.qubitpage.com/whitepaper},
+  note={Trained on AMD Instinct MI300X for the AMD Developer Hackathon}
+}
+```
+
+## Links
+
+- **Live Dashboard**: [sentinel.qubitpage.com](https://sentinel.qubitpage.com)
+- **Whitepaper**: [sentinel.qubitpage.com/whitepaper](https://sentinel.qubitpage.com/whitepaper)
+- **AMD Hackathon**: [lablab.ai/ai-hackathons/amd-developer](https://lablab.ai/ai-hackathons/amd-developer)

config.json

@@ -0,0 +1,27 @@
+{
+  "architectures": [
+    "SentinelBrainForCausalLM"
+  ],
+  "model_type": "sentinelbrain",
+  "vocab_size": 100277,
+  "hidden_size": 4096,
+  "num_hidden_layers": 24,
+  "num_attention_heads": 32,
+  "num_key_value_heads": 8,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 2048,
+  "rope_theta": 500000.0,
+  "rms_norm_eps": 1e-05,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "num_local_experts": 4,
+  "num_experts_per_tok": 2,
+  "max_experts": 256,
+  "routing_mode": "token_choice",
+  "expert_capacity_factor": 1.25,
+  "router_aux_loss_coeff": 0.01,
+  "router_z_loss_coeff": 0.001,
+  "training_step": 2471,
+  "training_val_loss": 1.9925728058815002,
+  "training_tokens": 178110464
+}

model-00001-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66e3102481a86dec86e0123c41752009cee8199c0e55d9f7fa9145fe3d2d1a5a
+size 5305517664

model-00002-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6cde9fb1113ae6429eab81d8435de19d6314b600f8ccde7ea3bb344b47edf4f1
+size 5289438728

model-00003-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c74d48ad85fb42bcd134726229a5026caa5ec253b820ef9461f5e737eb9a38ed
+size 5295647312

model-00004-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77ab32b99a87fd23b6ea872ce00b7ead4d523fc9e232797e124ed88e0be0fec4
+size 5289438840

model-00005-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29d2862b339d0df0bf5e89cb32d676326f702d1b7038124aac1696eaae637183
+size 5345987608

model-00006-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9105904112d62b3149a7a9f10251799c320751669c045f8605fc847e6ebea4ee
+size 3103339192

model.safetensors.index.json

@@ -0,0 +1,562 @@
+{
+  "metadata": {
+    "total_size": 29629309504
+  },
+  "weight_map": {
+    "tok_emb.weight": "model-00001-of-00006.safetensors",
+    "layers.0.attn_norm.weight": "model-00001-of-00006.safetensors",
+    "layers.0.attn.wq.weight": "model-00001-of-00006.safetensors",
+    "layers.0.attn.wk.weight": "model-00001-of-00006.safetensors",
+    "layers.0.attn.wv.weight": "model-00001-of-00006.safetensors",
+    "layers.0.attn.wo.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn_norm.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.expert_usage": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.expert_age": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.total_tokens_seen": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.router.gate.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.ec_router.gate.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.0.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.0.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.0.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.1.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.1.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.1.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.2.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.2.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.2.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.3.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.3.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.0.ffn.experts.3.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.1.attn_norm.weight": "model-00001-of-00006.safetensors",
+    "layers.1.attn.wq.weight": "model-00001-of-00006.safetensors",
+    "layers.1.attn.wk.weight": "model-00001-of-00006.safetensors",
+    "layers.1.attn.wv.weight": "model-00001-of-00006.safetensors",
+    "layers.1.attn.wo.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn_norm.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.expert_usage": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.expert_age": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.total_tokens_seen": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.router.gate.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.ec_router.gate.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.0.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.0.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.0.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.1.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.1.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.1.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.2.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.2.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.2.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.3.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.3.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.1.ffn.experts.3.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.2.attn_norm.weight": "model-00001-of-00006.safetensors",
+    "layers.2.attn.wq.weight": "model-00001-of-00006.safetensors",
+    "layers.2.attn.wk.weight": "model-00001-of-00006.safetensors",
+    "layers.2.attn.wv.weight": "model-00001-of-00006.safetensors",
+    "layers.2.attn.wo.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn_norm.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.expert_usage": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.expert_age": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.total_tokens_seen": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.router.gate.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.ec_router.gate.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.0.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.0.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.0.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.1.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.1.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.1.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.2.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.2.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.2.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.3.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.3.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.2.ffn.experts.3.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.3.attn_norm.weight": "model-00001-of-00006.safetensors",
+    "layers.3.attn.wq.weight": "model-00001-of-00006.safetensors",
+    "layers.3.attn.wk.weight": "model-00001-of-00006.safetensors",
+    "layers.3.attn.wv.weight": "model-00001-of-00006.safetensors",
+    "layers.3.attn.wo.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn_norm.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.expert_usage": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.expert_age": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.total_tokens_seen": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.router.gate.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.ec_router.gate.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.experts.0.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.experts.0.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.experts.0.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.experts.1.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.experts.1.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.experts.1.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.experts.2.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.experts.2.w_up.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.experts.2.w_down.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.experts.3.w_gate.weight": "model-00001-of-00006.safetensors",
+    "layers.3.ffn.experts.3.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.3.ffn.experts.3.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.4.attn_norm.weight": "model-00002-of-00006.safetensors",
+    "layers.4.attn.wq.weight": "model-00002-of-00006.safetensors",
+    "layers.4.attn.wk.weight": "model-00002-of-00006.safetensors",
+    "layers.4.attn.wv.weight": "model-00002-of-00006.safetensors",
+    "layers.4.attn.wo.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn_norm.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.expert_usage": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.expert_age": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.total_tokens_seen": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.router.gate.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.ec_router.gate.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.0.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.0.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.0.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.1.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.1.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.1.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.2.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.2.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.2.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.3.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.3.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.4.ffn.experts.3.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.5.attn_norm.weight": "model-00002-of-00006.safetensors",
+    "layers.5.attn.wq.weight": "model-00002-of-00006.safetensors",
+    "layers.5.attn.wk.weight": "model-00002-of-00006.safetensors",
+    "layers.5.attn.wv.weight": "model-00002-of-00006.safetensors",
+    "layers.5.attn.wo.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn_norm.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.expert_usage": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.expert_age": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.total_tokens_seen": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.router.gate.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.ec_router.gate.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.0.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.0.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.0.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.1.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.1.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.1.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.2.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.2.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.2.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.3.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.3.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.5.ffn.experts.3.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.6.attn_norm.weight": "model-00002-of-00006.safetensors",
+    "layers.6.attn.wq.weight": "model-00002-of-00006.safetensors",
+    "layers.6.attn.wk.weight": "model-00002-of-00006.safetensors",
+    "layers.6.attn.wv.weight": "model-00002-of-00006.safetensors",
+    "layers.6.attn.wo.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn_norm.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.expert_usage": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.expert_age": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.total_tokens_seen": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.router.gate.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.ec_router.gate.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.0.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.0.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.0.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.1.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.1.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.1.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.2.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.2.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.2.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.3.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.3.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.6.ffn.experts.3.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.7.attn_norm.weight": "model-00002-of-00006.safetensors",
+    "layers.7.attn.wq.weight": "model-00002-of-00006.safetensors",
+    "layers.7.attn.wk.weight": "model-00002-of-00006.safetensors",
+    "layers.7.attn.wv.weight": "model-00002-of-00006.safetensors",
+    "layers.7.attn.wo.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn_norm.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.expert_usage": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.expert_age": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.total_tokens_seen": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.router.gate.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.ec_router.gate.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.0.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.0.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.0.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.1.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.1.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.1.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.2.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.2.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.2.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.3.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.3.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.7.ffn.experts.3.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.8.attn_norm.weight": "model-00002-of-00006.safetensors",
+    "layers.8.attn.wq.weight": "model-00002-of-00006.safetensors",
+    "layers.8.attn.wk.weight": "model-00002-of-00006.safetensors",
+    "layers.8.attn.wv.weight": "model-00002-of-00006.safetensors",
+    "layers.8.attn.wo.weight": "model-00002-of-00006.safetensors",
+    "layers.8.ffn_norm.weight": "model-00002-of-00006.safetensors",
+    "layers.8.ffn.expert_usage": "model-00002-of-00006.safetensors",
+    "layers.8.ffn.expert_age": "model-00002-of-00006.safetensors",
+    "layers.8.ffn.total_tokens_seen": "model-00002-of-00006.safetensors",
+    "layers.8.ffn.router.gate.weight": "model-00002-of-00006.safetensors",
+    "layers.8.ffn.ec_router.gate.weight": "model-00002-of-00006.safetensors",
+    "layers.8.ffn.experts.0.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.8.ffn.experts.0.w_up.weight": "model-00002-of-00006.safetensors",
+    "layers.8.ffn.experts.0.w_down.weight": "model-00002-of-00006.safetensors",
+    "layers.8.ffn.experts.1.w_gate.weight": "model-00002-of-00006.safetensors",
+    "layers.8.ffn.experts.1.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.8.ffn.experts.1.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.8.ffn.experts.2.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.8.ffn.experts.2.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.8.ffn.experts.2.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.8.ffn.experts.3.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.8.ffn.experts.3.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.8.ffn.experts.3.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.9.attn_norm.weight": "model-00003-of-00006.safetensors",
+    "layers.9.attn.wq.weight": "model-00003-of-00006.safetensors",
+    "layers.9.attn.wk.weight": "model-00003-of-00006.safetensors",
+    "layers.9.attn.wv.weight": "model-00003-of-00006.safetensors",
+    "layers.9.attn.wo.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn_norm.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.expert_usage": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.expert_age": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.total_tokens_seen": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.router.gate.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.ec_router.gate.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.0.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.0.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.0.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.1.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.1.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.1.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.2.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.2.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.2.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.3.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.3.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.9.ffn.experts.3.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.10.attn_norm.weight": "model-00003-of-00006.safetensors",
+    "layers.10.attn.wq.weight": "model-00003-of-00006.safetensors",
+    "layers.10.attn.wk.weight": "model-00003-of-00006.safetensors",
+    "layers.10.attn.wv.weight": "model-00003-of-00006.safetensors",
+    "layers.10.attn.wo.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn_norm.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.expert_usage": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.expert_age": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.total_tokens_seen": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.router.gate.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.ec_router.gate.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.0.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.0.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.0.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.1.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.1.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.1.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.2.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.2.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.2.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.3.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.3.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.10.ffn.experts.3.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.11.attn_norm.weight": "model-00003-of-00006.safetensors",
+    "layers.11.attn.wq.weight": "model-00003-of-00006.safetensors",
+    "layers.11.attn.wk.weight": "model-00003-of-00006.safetensors",
+    "layers.11.attn.wv.weight": "model-00003-of-00006.safetensors",
+    "layers.11.attn.wo.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn_norm.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.expert_usage": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.expert_age": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.total_tokens_seen": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.router.gate.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.ec_router.gate.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.0.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.0.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.0.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.1.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.1.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.1.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.2.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.2.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.2.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.3.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.3.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.11.ffn.experts.3.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.12.attn_norm.weight": "model-00003-of-00006.safetensors",
+    "layers.12.attn.wq.weight": "model-00003-of-00006.safetensors",
+    "layers.12.attn.wk.weight": "model-00003-of-00006.safetensors",
+    "layers.12.attn.wv.weight": "model-00003-of-00006.safetensors",
+    "layers.12.attn.wo.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn_norm.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.expert_usage": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.expert_age": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.total_tokens_seen": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.router.gate.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.ec_router.gate.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.0.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.0.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.0.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.1.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.1.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.1.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.2.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.2.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.2.w_down.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.3.w_gate.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.3.w_up.weight": "model-00003-of-00006.safetensors",
+    "layers.12.ffn.experts.3.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.13.attn_norm.weight": "model-00004-of-00006.safetensors",
+    "layers.13.attn.wq.weight": "model-00004-of-00006.safetensors",
+    "layers.13.attn.wk.weight": "model-00004-of-00006.safetensors",
+    "layers.13.attn.wv.weight": "model-00004-of-00006.safetensors",
+    "layers.13.attn.wo.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn_norm.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.expert_usage": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.expert_age": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.total_tokens_seen": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.router.gate.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.ec_router.gate.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.0.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.0.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.0.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.1.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.1.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.1.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.2.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.2.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.2.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.3.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.3.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.13.ffn.experts.3.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.14.attn_norm.weight": "model-00004-of-00006.safetensors",
+    "layers.14.attn.wq.weight": "model-00004-of-00006.safetensors",
+    "layers.14.attn.wk.weight": "model-00004-of-00006.safetensors",
+    "layers.14.attn.wv.weight": "model-00004-of-00006.safetensors",
+    "layers.14.attn.wo.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn_norm.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.expert_usage": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.expert_age": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.total_tokens_seen": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.router.gate.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.ec_router.gate.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.0.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.0.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.0.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.1.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.1.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.1.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.2.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.2.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.2.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.3.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.3.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.14.ffn.experts.3.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.15.attn_norm.weight": "model-00004-of-00006.safetensors",
+    "layers.15.attn.wq.weight": "model-00004-of-00006.safetensors",
+    "layers.15.attn.wk.weight": "model-00004-of-00006.safetensors",
+    "layers.15.attn.wv.weight": "model-00004-of-00006.safetensors",
+    "layers.15.attn.wo.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn_norm.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.expert_usage": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.expert_age": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.total_tokens_seen": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.router.gate.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.ec_router.gate.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.0.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.0.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.0.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.1.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.1.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.1.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.2.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.2.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.2.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.3.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.3.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.15.ffn.experts.3.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.16.attn_norm.weight": "model-00004-of-00006.safetensors",
+    "layers.16.attn.wq.weight": "model-00004-of-00006.safetensors",
+    "layers.16.attn.wk.weight": "model-00004-of-00006.safetensors",
+    "layers.16.attn.wv.weight": "model-00004-of-00006.safetensors",
+    "layers.16.attn.wo.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn_norm.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.expert_usage": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.expert_age": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.total_tokens_seen": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.router.gate.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.ec_router.gate.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.0.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.0.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.0.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.1.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.1.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.1.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.2.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.2.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.2.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.3.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.3.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.16.ffn.experts.3.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.17.attn_norm.weight": "model-00004-of-00006.safetensors",
+    "layers.17.attn.wq.weight": "model-00004-of-00006.safetensors",
+    "layers.17.attn.wk.weight": "model-00004-of-00006.safetensors",
+    "layers.17.attn.wv.weight": "model-00004-of-00006.safetensors",
+    "layers.17.attn.wo.weight": "model-00004-of-00006.safetensors",
+    "layers.17.ffn_norm.weight": "model-00004-of-00006.safetensors",
+    "layers.17.ffn.expert_usage": "model-00004-of-00006.safetensors",
+    "layers.17.ffn.expert_age": "model-00004-of-00006.safetensors",
+    "layers.17.ffn.total_tokens_seen": "model-00004-of-00006.safetensors",
+    "layers.17.ffn.router.gate.weight": "model-00004-of-00006.safetensors",
+    "layers.17.ffn.ec_router.gate.weight": "model-00004-of-00006.safetensors",
+    "layers.17.ffn.experts.0.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.17.ffn.experts.0.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.17.ffn.experts.0.w_down.weight": "model-00004-of-00006.safetensors",
+    "layers.17.ffn.experts.1.w_gate.weight": "model-00004-of-00006.safetensors",
+    "layers.17.ffn.experts.1.w_up.weight": "model-00004-of-00006.safetensors",
+    "layers.17.ffn.experts.1.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.17.ffn.experts.2.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.17.ffn.experts.2.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.17.ffn.experts.2.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.17.ffn.experts.3.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.17.ffn.experts.3.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.17.ffn.experts.3.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.18.attn_norm.weight": "model-00005-of-00006.safetensors",
+    "layers.18.attn.wq.weight": "model-00005-of-00006.safetensors",
+    "layers.18.attn.wk.weight": "model-00005-of-00006.safetensors",
+    "layers.18.attn.wv.weight": "model-00005-of-00006.safetensors",
+    "layers.18.attn.wo.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn_norm.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.expert_usage": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.expert_age": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.total_tokens_seen": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.router.gate.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.ec_router.gate.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.0.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.0.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.0.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.1.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.1.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.1.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.2.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.2.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.2.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.3.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.3.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.18.ffn.experts.3.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.19.attn_norm.weight": "model-00005-of-00006.safetensors",
+    "layers.19.attn.wq.weight": "model-00005-of-00006.safetensors",
+    "layers.19.attn.wk.weight": "model-00005-of-00006.safetensors",
+    "layers.19.attn.wv.weight": "model-00005-of-00006.safetensors",
+    "layers.19.attn.wo.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn_norm.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.expert_usage": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.expert_age": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.total_tokens_seen": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.router.gate.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.ec_router.gate.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.0.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.0.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.0.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.1.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.1.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.1.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.2.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.2.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.2.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.3.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.3.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.19.ffn.experts.3.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.20.attn_norm.weight": "model-00005-of-00006.safetensors",
+    "layers.20.attn.wq.weight": "model-00005-of-00006.safetensors",
+    "layers.20.attn.wk.weight": "model-00005-of-00006.safetensors",
+    "layers.20.attn.wv.weight": "model-00005-of-00006.safetensors",
+    "layers.20.attn.wo.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn_norm.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.expert_usage": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.expert_age": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.total_tokens_seen": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.router.gate.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.ec_router.gate.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.0.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.0.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.0.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.1.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.1.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.1.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.2.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.2.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.2.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.3.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.3.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.20.ffn.experts.3.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.21.attn_norm.weight": "model-00005-of-00006.safetensors",
+    "layers.21.attn.wq.weight": "model-00005-of-00006.safetensors",
+    "layers.21.attn.wk.weight": "model-00005-of-00006.safetensors",
+    "layers.21.attn.wv.weight": "model-00005-of-00006.safetensors",
+    "layers.21.attn.wo.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn_norm.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.expert_usage": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.expert_age": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.total_tokens_seen": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.router.gate.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.ec_router.gate.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.0.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.0.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.0.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.1.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.1.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.1.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.2.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.2.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.2.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.3.w_gate.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.3.w_up.weight": "model-00005-of-00006.safetensors",
+    "layers.21.ffn.experts.3.w_down.weight": "model-00005-of-00006.safetensors",
+    "layers.22.attn_norm.weight": "model-00005-of-00006.safetensors",
+    "layers.22.attn.wq.weight": "model-00005-of-00006.safetensors",
+    "layers.22.attn.wk.weight": "model-00005-of-00006.safetensors",
+    "layers.22.attn.wv.weight": "model-00005-of-00006.safetensors",
+    "layers.22.attn.wo.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn_norm.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.expert_usage": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.expert_age": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.total_tokens_seen": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.router.gate.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.ec_router.gate.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.0.w_gate.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.0.w_up.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.0.w_down.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.1.w_gate.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.1.w_up.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.1.w_down.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.2.w_gate.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.2.w_up.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.2.w_down.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.3.w_gate.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.3.w_up.weight": "model-00006-of-00006.safetensors",
+    "layers.22.ffn.experts.3.w_down.weight": "model-00006-of-00006.safetensors",
+    "layers.23.attn_norm.weight": "model-00006-of-00006.safetensors",
+    "layers.23.attn.wq.weight": "model-00006-of-00006.safetensors",
+    "layers.23.attn.wk.weight": "model-00006-of-00006.safetensors",
+    "layers.23.attn.wv.weight": "model-00006-of-00006.safetensors",
+    "layers.23.attn.wo.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn_norm.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.expert_usage": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.expert_age": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.total_tokens_seen": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.router.gate.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.ec_router.gate.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.0.w_gate.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.0.w_up.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.0.w_down.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.1.w_gate.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.1.w_up.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.1.w_down.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.2.w_gate.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.2.w_up.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.2.w_down.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.3.w_gate.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.3.w_up.weight": "model-00006-of-00006.safetensors",
+    "layers.23.ffn.experts.3.w_down.weight": "model-00006-of-00006.safetensors",
+    "norm.weight": "model-00006-of-00006.safetensors",
+    "lm_head.weight": "model-00006-of-00006.safetensors"
+  }
+}

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+  "model_type": "tiktoken",
+  "encoding_name": "cl100k_base",
+  "vocab_size": 100277,
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>"
+}