Super-squash branch 'main' using huggingface_hub

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,250 @@
+---
+library_name: transformers
+license: apache-2.0
+pipeline_tag: text-generation
+tags:
+  - hybrid
+  - ssm
+  - state-space-model
+  - linear-attention
+  - mamba-2
+  - priming
+  - long-context
+  - instruction-tuned
+base_model: Qwen/Qwen3-8B
+paper:
+  - https://arxiv.org/abs/2405.21060
+---
+
+# Mamba2-primed-HQwen3-8B-Instruct
+
+Mamba2-primed-HQwen3-8B-Instruct is a Hybrid language model consisting of 50% Attention layers and 50% [Mamba2](https://arxiv.org/abs/2405.21060) layers, primed from [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) using the [Hybrid Model Factory](https://github.com/awslabs/hybrid-model-factory) Priming pipeline. The model is instruction-tuned and supports context lengths up to 128K tokens.
+
+Mamba-2 is a State-Space Model layer with constant memory and linear compute cost in the sequence length.
+
+By combining Attention with Mamba-2, our Hybrid model achieves up to **2× faster inference** at long contexts while **closely matching the base Transformer's quality**.
+
+## Links
+
+- 📄 [Mamba-2 paper](https://arxiv.org/abs/2405.21060)
+- 💻 [GitHub: Hybrid Model Factory](https://github.com/awslabs/hybrid-model-factory)
+
+
+
+## Why Hybrid?
+
+Each Primed Hybrid model is initialized from a base Transformer by converting a portion of its Attention layers into State-Space Model (SSM) layers that maintain a fixed-size recurrent state instead of a growing KV cache. At a 50% Hybrid ratio, roughly half the KV cache (which grows linearly with sequence length) is replaced with fixed-size SSM state. The practical benefits:
+
+- **Higher throughput at long contexts** — less memory on KV cache means more memory for batching
+- **More concurrent sequences** — ~2× as many concurrent sequences before hitting memory limits
+- **Growing advantage with context length** — at long contexts, Attention dominates the forward pass while SSM layers remain negligible in cost. Since the Hybrid model makes roughly half as many Attention calls as the base Transformer, the throughput advantage grows with context length
+
+Increasing hybridization ratio, replacing more Attention layers with SSM layers, further reduces memory and increases throughput, typically at the expense of performance.
+
+
+## Model Overview
+
+- **Type**: Causal Language Model (Hybrid Attention + SSM)
+- **Base Model**: Qwen3-8B
+- **Hybrid Layer Type**: Mamba-2
+- **Hybrid Ratio**: 50% (18 Attention + 18 Mamba-2 layers)
+- **Parameters**: ~8B
+- **Context Length**: 128K natively
+- **Precision**: bfloat16
+- **License**: Apache 2.0
+
+
+Note, this is an Instruct-tuned model and is not a thinking model, that is, it does not natively produce chain-of-thought thinking tokens in its generation trace.
+
+## Benchmark Results
+
+Below we report benchmark performance for all our instruct-tuned Primed models. All Hybrid models use a 50% Hybrid ratio and are Primed from Qwen3-8B. 
+
+We consider two baselines:
+
+1. **Qwen3-8B (non-thinking, from HF)**: The original Qwen model evaluated in non-thinking mode, which is the intended mode for an Instruct model. This serves as the base Transformer from which we start the Priming procedure.
+2. **Qwen3-8B (Long)**: The Qwen model fine-tuned on our priming data, extending its native context length from 32K to 128K. All Primed Hybrid models use the same training hyperparameters and data as this baseline, making it a fair comparison for differing architectures.
+
+On both long- and short-context benchmarks, our Primed Hybrid models closely match the performance of the Transformer model while having [considerably lower deployment costs](#inference-efficiency), showcasing the efficacy of the Priming process.
+
+### Long-Context Benchmarks
+
+Evaluated on [HELMET](https://github.com/princeton-nlp/HELMET), [MRCR](https://huggingface.co/datasets/openai/mrcr), and [BABILong](https://github.com/booydar/babilong) across context lengths from 8K to 128K, using a weighted average with geometrically increasing weights for longer contexts. 
+
+The plot below shows performance averaged over context lengths from 8K to 128K.
+
+**Note.** For the Qwen3-8B (non-thinking, from HF) model, we used YaRN to evaluate on long-context tasks as directed in the [model card](https://huggingface.co/Qwen/Qwen3-8B)
+
+<img src="https://github.com/awslabs/hybrid-model-factory/blob/main/assets/figures/long_context_results_8B_models_final.png" title="Long Context Results 8B Models" />
+
+
+**How close are the Hybrid models to the Transformer baseline on long context tasks?**
+Primed GKA and GDN Hybrids are within ~1.5 points of Qwen3-8B (Long) on average, while being [1.5–2× faster at inference](#inference-efficiency). Primed BMOJO-F matches GKA/GDN in quality but is slower due to unfused SSM+SWA kernels ([details](https://github.com/awslabs/hybrid-model-factory/blob/main/docs/Inference.md)). Primed Mamba2 lags further behind (~3 point gap), consistent with GKA and GDN's higher expressivity.
+
+**Why SSM layers over Sliding Window Attention (SWA)?**
+All Hybrid SSM models outperform the Hybrid SWA model (50% Attention + 50% SWA, window size 512). Even though SWA uses ~2× the effective state size of GKA at BF16, SSM layers retain information from the remote past, while SWA forgets everything beyond its window.
+
+### Short-Context NLP Benchmarks
+
+Evaluations on Tulu3-dev from [OLMES](https://github.com/allenai/olmes). All tasks are over a short-context length (≤ 8K).
+Each category in the table below averages the following Tulu3-dev subtasks: 
+1. Math: GSM8K, MATH,
+2. Knowledge: MMLU, PopQA, TruthfulQA,
+3. Coding: HumanEval, HumanEval+, 
+4. Reasoning: BigBenchHard,
+5. Instruction Following: IFEval.
+
+|Model | Math | Knowledge | Coding | Reasoning | Instruction Following  | Average |
+|---|---|---|---|----------|---|--------|
+| Qwen3-8B [non-thinking, from HF] | 81.36 | 49.33 | 91.77 | 74.31 | 85.59 | 76.47  |
+| Qwen3-8B [Long] | 64.56 | 49.75 | 91 | 76.27    | 74.49  | 71.21  |
+| GKA-primed-HQwen3-8B-Instruct    |  64.15 | 47.90 | 90.46 | 72.60    | 70.98 | 69.22  |
+| GDN-primed-HQwen3-8B-Instruct    |  59.54 | 48.41 | 91.18 | 72.97    | 73.57 | 69.13  |
+| Mamba2-primed-HQwen3-8B-Instruct | 57.77 | 46.91 | 89.56 | 70.99    | 74.86 | 68.02  |
+| BMOJOF-primed-HQwen3-8B-Instruct | 65.69 | 48.63 | 90.02 | 76.42    | 75.60 | 71.27  |
+
+
+**How close are the Hybrid models to the Transformer baseline on short context tasks?**
+All Primed Hybrid models are within ~3 points of Qwen3-8B (Long), using [< 0.5% of the base Transformer's pre-training token budget](#training-data). Note, BMOJO-F [w/ GKA] fully matches the Transformer baseline but is slower to deploy (see above).
+
+**Which SSM layer type performs best?**
+Among the non BMOJO-F Hybrids, GKA ranks first (~2 point gap with Qwen3-8B [Long]), followed by GDN, then Mamba2. This ranking correlates with the expressiveness order of their respective SSM layers.
+
+## About Mamba-2
+
+Mamba-2 is a State-Space Model layer with diagonal transition dynamics and input-dependent gating. It processes sequences in linear time with constant memory, making it efficient for long-context inference. Mamba-2 is less expressive than Gated DeltaNet and Gated KalmaNet due to its diagonal structure, but benefits from well-optimized kernels.
+
+For more details, see the [Mamba-2 paper](https://arxiv.org/abs/2405.21060).
+
+
+
+### Architecture Details
+
+| Component | Details                                                                      |
+|-----------|------------------------------------------------------------------------------|
+| Number of Layers | 36 (18 Attention + 18 Mamba-2)                                                   |
+| Hidden Dimension | 4096                                                                         |
+| Attention Heads | 32 (Q) / 8 (KV)                                                              |
+| Head Dimension | 128                                                                          |
+| Intermediate Dimension (FFN) | 12288                                                                        |
+| Vocabulary Size | 151,936                                                                      |
+| Position Encoding | RoPE (θ = 5,000,000)                                                         |
+| Layer Layout | Mamba-2 layer indices were selected with our [*selective hybridization*](https://github.com/awslabs/hybrid-model-factory/blob/main/docs/LayerSelection.md) procedure |
+
+
+
+
+### Inference Efficiency
+
+Sustained decode throughput (tokens/s) on 8× H200 GPUs (TP=8), measured during pure decode with a saturated KV cache. Benchmarked with random data (no prefix-caching benefits). See the full [Inference guide](https://github.com/awslabs/hybrid-model-factory/blob/main/docs/Inference.md#performance-benchmarks) for methodology and additional models.
+
+| Model                              | 16K            | 32K            | 64K            | 128K           |
+|------------------------------------|----------------|----------------|----------------|----------------|
+| Mamba2-primed-HQwen3-8B-Instruct  | 16,844 (1.88×) | 9,966 (1.93×)  | 5,460 (1.99×)  | 2,825 (2.30×)  |
+| GKA-primed-HQwen3-8B              | 15,892 (1.78×) | 9,159 (1.77×)  | 5,173 (1.89×)  | 2,736 (2.23×)  |
+| GDN-primed-HQwen3-8B              | 17,479 (1.95×) | 10,080 (1.95×) | 5,521 (2.01×)  | 2,863 (2.33×)  |
+| BMOJOF-primed-HQwen3-8B           | 7,854 (0.88×)  | 5,597 (1.08×)  | 3,573 (1.30×)  | 2,153 (1.75×)  |
+| Qwen3-8B (Long)                   | 8,951          | 5,174          | 2,740          | 1,227          |
+
+Mean TTFT at the Transformer's saturated batch size (Hybrid model has memory to spare):
+
+| Model                              | 16K              | 32K              | 64K              | 128K             |
+|------------------------------------|------------------|------------------|------------------|------------------|
+| Mamba2-primed-HQwen3-8B-Instruct  | 28,668 ms (1.03×) | 31,405 ms (0.96×) | 36,666 ms (0.86×) | 46,618 ms (0.74×) |
+| GKA-primed-HQwen3-8B              | 35,013 ms (1.26×) | 38,502 ms (1.18×) | 44,893 ms (1.06��) | 53,606 ms (0.85×) |
+| GDN-primed-HQwen3-8B              | 27,805 ms (1.00×) | 30,975 ms (0.95×) | 36,151 ms (0.85×) | 46,389 ms (0.74×) |
+| BMOJOF-primed-HQwen3-8B           | 44,763 ms (1.61×) | 47,600 ms (1.46×) | 52,272 ms (1.23×) | 61,702 ms (0.98×) |
+| Qwen3-8B (Long)                   | 27,736 ms        | 32,661 ms        | 42,462 ms        | 62,922 ms        |
+
+The decode throughput advantage grows with context length — from 1.88× at 16K to 2.30× at 128K — thanks to Mamba2 layers maintaining a fixed-size recurrent state instead of a growing KV cache. TTFT crosses over at 32K and reaches 0.74× (26% faster) at 128K.
+
+
+
+
+## Usage
+
+### With vLLM (recommended)
+
+Install the [Hybrid Model Factory vLLM plugin](https://github.com/awslabs/hybrid-model-factory/blob/main/docs/Inference.md#docker-recommended) in your local environment, then serve:
+
+```bash
+vllm serve amazon/Mamba2-primed-HQwen3-8B-Instruct \
+  --enable-prefix-caching \
+  --mamba-cache-mode align \
+  --mamba-cache-dtype float32 \
+  --mamba-ssm-cache-dtype float32
+```
+
+Query the server:
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "amazon/Mamba2-primed-HQwen3-8B-Instruct",
+    "messages": [
+      {"role": "system", "content": "You are a helpful assistant."},
+      {"role": "user", "content": "What is Linear Attention in the context of LLMs?"}
+    ]
+  }'
+```
+
+> **Tip:** The `--mamba-cache-dtype float32` and `--mamba-ssm-cache-dtype float32` flags are important for accurate long-context generation. See the [Inference guide](https://github.com/awslabs/hybrid-model-factory/blob/main/docs/Inference.md#recommended-flags-for-hybrid-models) for details on all recommended flags.
+
+### With HuggingFace Transformers
+See the [Inference guide](https://github.com/awslabs/hybrid-model-factory/blob/main/docs/Inference.md#recommended-flags-for-hybrid-models) for details on when we recommend the HuggingFace Transformers implementation as opposed to the highly optimized vLLM one.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import hmf.model.hybrid_zoo.models.model_register  # Register Hybrid models
+
+model = AutoModelForCausalLM.from_pretrained(
+    "amazon/Mamba2-primed-HQwen3-8B-Instruct", trust_remote_code=True
+).to("cuda")
+tokenizer = AutoTokenizer.from_pretrained("amazon/Mamba2-primed-HQwen3-8B-Instruct")
+
+messages = [{"role": "user", "content": "What is linear attention in the context of LLMs?"}]
+prompt = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True
+)
+inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+outputs = model.generate(**inputs, max_new_tokens=256)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+### Training-Free Context Extension 
+
+This model supports training-free context extension 2-4× its native context via an extension to Hybrid models of [PICASO cache composition](https://arxiv.org/abs/2502.17605). See the [State Composition guide](https://github.com/awslabs/hybrid-model-factory/blob/main/docs/StateComposition.md) for usage. Note, this is currently only supported in HF for now.
+
+
+## Training data
+These models were produced through the multi-stage Priming pipeline from [Hybrid Model Factory](https://github.com/awslabs/hybrid-model-factory). Training data spans web documents, mathematics, long-context documents, and instruction-following and reasoning examples — each targeting a different capability axis. This diversity is critical: it allows the Priming procedure to convert a base Transformer into a more memory- and compute-efficient Hybrid architecture at nearly the same level of performance, using < 0.5% of the base Transformer model's pre-training token budget.
+
+## Responsible AI Considerations
+At Amazon, we are committed to developing AI responsibly and take a people-centric approach that prioritizes education, science, and our customers, to integrate responsible AI across the end-to-end AI lifecycle. We believe the use of AI must respect the rule of law and human rights, and we encourage the safe and responsible development of AI.  When downloaded or used in accordance with [AWS Responsible AI Policy](https://aws.amazon.com/ai/responsible-ai/policy/), developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+Please report model quality, risk, security vulnerabilities or Amazon AI Concerns [here](https://pages.awscloud.com/global-ln-gc-400-ai-service-cards-contact-us-registration.html).
+
+## Citation
+
+```bibtex
+@software{hybrid_model_factory,
+  title = {Hybrid Model Factory},
+  year = {2026},
+  url = {https://github.com/awslabs/hybrid-model-factory}
+}
+
+@misc{dao2024transformersssmsgeneralizedmodels,
+      title={Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality}, 
+      author={Tri Dao and Albert Gu},
+      year={2024},
+      eprint={2405.21060},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2405.21060}, 
+}
+
+```
+
+## License
+
+This model is licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).

chat_template.jinja

@@ -0,0 +1,87 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+        {{- '<think>\n\n</think>\n\n' }}
+{%- endif %}

config.json

@@ -0,0 +1,88 @@
+{
+  "architectures": [
+    "HybridQwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "hybrid_override_pattern": "*-M2-M2-M2-M2-M2-M2-*-*-M2-M2-M2-*-*-*-*-*-*-*-*-*-*-*-*-*-*-M2-M2-M2-*-M2-M2-M2-M2-M2-M2",
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "layer_types": [
+    "*",
+    "M2",
+    "M2",
+    "M2",
+    "M2",
+    "M2",
+    "M2",
+    "*",
+    "*",
+    "M2",
+    "M2",
+    "M2",
+    "*",
+    "*",
+    "*",
+    "*",
+    "*",
+    "*",
+    "*",
+    "*",
+    "*",
+    "*",
+    "*",
+    "*",
+    "*",
+    "*",
+    "M2",
+    "M2",
+    "M2",
+    "*",
+    "M2",
+    "M2",
+    "M2",
+    "M2",
+    "M2",
+    "M2"
+  ],
+  "mamba2_config": {
+    "d_inner": 4096,
+    "d_model": 4096,
+    "d_xb": 1024,
+    "hidden_act": "silu",
+    "intermediate_size": 12288,
+    "n_layer": 36,
+    "rms_norm_eps": 1e-06,
+    "ssm_cfg": {
+      "d_state": 128,
+      "expand": 1,
+      "ngroups": 32
+    },
+    "use_pos_emb": false,
+    "use_qk_norm": true
+  },
+  "max_position_embeddings": 131072,
+  "max_window_layers": 36,
+  "model_type": "hybrid_qwen3",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 36,
+  "num_key_value_heads": 8,
+  "pad_token_id": null,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 5000000,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.3.0",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "transformers_version": "4.51.3"
+}

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d599114db34a8209c3a4db4131abab5f022729e05e7f20272b0d5844db7581d
+size 4921044208

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b46584cc12533ae662dc38547db55f0329c0575e2acd47188a86987d0c612b51
+size 4916923920

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94efbe687ff50d4c3114b617279674ae9b86d51ff8b9a557e53c640cb5c17c17
+size 4951506888

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:509f87004edbb1ebdb914a344edcaf25f0fa0c4fefd780736f6a806faadee9fb
+size 2202003496

model.safetensors.index.json

@@ -0,0 +1,478 @@
+{
+  "metadata": {
+    "total_size": 16991425920
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00004-of-00004.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.B_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.C_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.dt_bias": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.A_log": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.B_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.C_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.D": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.dt_bias": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.A_log": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.B_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.C_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.D": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.dt_bias": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.B_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.C_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.dt_bias": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.B_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.C_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.dt_bias": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.27.mamba.B_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mamba.C_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.27.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.27.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mamba.dt_bias": "model-00003-of-00004.safetensors",
+    "model.layers.27.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mamba.norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.28.mamba.B_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mamba.C_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.28.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.28.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mamba.dt_bias": "model-00003-of-00004.safetensors",
+    "model.layers.28.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mamba.norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.B_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.C_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.dt_bias": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.30.mamba.B_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mamba.C_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.30.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.30.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mamba.dt_bias": "model-00003-of-00004.safetensors",
+    "model.layers.30.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mamba.norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.31.mamba.B_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mamba.C_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.31.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.31.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mamba.dt_bias": "model-00003-of-00004.safetensors",
+    "model.layers.31.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mamba.norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.32.mamba.B_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mamba.C_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.32.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.32.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mamba.dt_bias": "model-00003-of-00004.safetensors",
+    "model.layers.32.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mamba.norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.33.mamba.B_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.33.mamba.C_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.33.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.33.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
+    "model.layers.33.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
+    "model.layers.33.mamba.dt_bias": "model-00003-of-00004.safetensors",
+    "model.layers.33.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.33.mamba.norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.33.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.34.mamba.A_log": "model-00004-of-00004.safetensors",
+    "model.layers.34.mamba.B_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.34.mamba.C_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.34.mamba.D": "model-00004-of-00004.safetensors",
+    "model.layers.34.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
+    "model.layers.34.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
+    "model.layers.34.mamba.dt_bias": "model-00004-of-00004.safetensors",
+    "model.layers.34.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.34.mamba.norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.34.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mamba.A_log": "model-00004-of-00004.safetensors",
+    "model.layers.35.mamba.B_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mamba.C_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mamba.D": "model-00004-of-00004.safetensors",
+    "model.layers.35.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
+    "model.layers.35.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mamba.dt_bias": "model-00004-of-00004.safetensors",
+    "model.layers.35.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mamba.norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.4.mamba.B_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mamba.C_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.4.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.4.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mamba.dt_bias": "model-00001-of-00004.safetensors",
+    "model.layers.4.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mamba.norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.B_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.C_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.dt_bias": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.B_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.C_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.dt_bias": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.mamba.A_log": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.B_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.C_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.D": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.dt_bias": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.norm.weight": "model-00004-of-00004.safetensors"
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "is_local": true,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}