Add files using upload-large-folder tool

Qwen3-ASR-1.7B/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

Qwen3-ASR-1.7B/README.md

@@ -0,0 +1,1393 @@
+---
+license: apache-2.0
+pipeline_tag: automatic-speech-recognition
+---
+
+# Qwen3-ASR
+
+## Overview
+
+### Introduction
+
+<p align="center">
+    <img src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/qwen3_asr_introduction.png" width="90%"/>
+<p>
+
+The Qwen3-ASR family includes Qwen3-ASR-1.7B and Qwen3-ASR-0.6B, which support language identification and ASR for 52 languages and dialects. Both leverage large-scale speech training data and the strong audio understanding capability of their foundation model, Qwen3-Omni. Experiments show that the 1.7B version achieves state-of-the-art performance among open-source ASR models and is competitive with the strongest proprietary commercial APIs. Here are the main features:
+
+* **All-in-one**: Qwen3-ASR-1.7B and Qwen3-ASR-0.6B support language identification and speech recognition for 30 languages and 22 Chinese dialects, so as to English accents from multiple countries and regions.
+
+* **Excellent and Fast**: The Qwen3-ASR family ASR models maintains high-quality and robust recognition under complex acoustic environments and challenging text patterns. Qwen3-ASR-1.7B achieves strong performance on both open-sourced and internal benchmarks. While the 0.6B version achieves accuracy-efficient trade-off, it reaches 2000 times throughput at a concurrency of 128. They both achieve streaming / offline unified inference with single model and support transcribe long audio.
+
+* **Novel and strong forced alignment Solution**: We introduce Qwen3-ForcedAligner-0.6B, which supports timestamp prediction for arbitrary units within up to 5 minutes of speech in 11 languages. Evaluations show its timestamp accuracy surpasses E2E based forced-alignment models.
+
+* **Comprehensive inference toolkit**: In addition to open-sourcing the architectures and weights of the Qwen3-ASR series, we also release a powerful, full-featured inference framework that supports vLLM-based batch inference, asynchronous serving, streaming inference, timestamp prediction, and more.
+
+### Model Architecture
+
+<p align="center">
+    <img src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/overview.jpg" width="100%"/>
+<p>
+
+
+### Released Models Description and Download
+
+Below is an introduction and download information for the Qwen3-ASR models. Please select and download the model that fits your needs.
+
+| Model | Supported Languages | Supported Dialects | Inference Mode | Audio Types |
+|---|---|---|---|---|
+| Qwen3-ASR-1.7B & Qwen3-ASR-0.6B | Chinese (zh), English (en), Cantonese (yue), Arabic (ar), German (de), French (fr), Spanish (es), Portuguese (pt), Indonesian (id), Italian (it), Korean (ko), Russian (ru), Thai (th), Vietnamese (vi), Japanese (ja), Turkish (tr), Hindi (hi), Malay (ms), Dutch (nl), Swedish (sv), Danish (da), Finnish (fi), Polish (pl), Czech (cs), Filipino (fil), Persian (fa), Greek (el), Hungarian (hu), Macedonian (mk), Romanian (ro) | Anhui, Dongbei, Fujian, Gansu, Guizhou, Hebei, Henan, Hubei, Hunan, Jiangxi, Ningxia, Shandong, Shaanxi, Shanxi, Sichuan, Tianjin, Yunnan, Zhejiang, Cantonese (Hong Kong accent), Cantonese (Guangdong accent), Wu language, Minnan language. | Offline / Streaming | Speech, Singing Voice, Songs with BGM |
+| Qwen3-ForcedAligner-0.6B | Chinese, English, Cantonese, French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish | -- | NAR | Speech |
+
+During model loading in the `qwen-asr` package or vLLM, model weights will be downloaded automatically based on the model name. However, if your runtime environment does not allow downloading weights during execution, you can use the following commands to manually download the model weights to a local directory:
+
+```bash
+# Download through ModelScope (recommended for users in Mainland China)
+pip install -U modelscope
+modelscope download --model Qwen/Qwen3-ASR-1.7B  --local_dir ./Qwen3-ASR-1.7B
+modelscope download --model Qwen/Qwen3-ASR-0.6B --local_dir ./Qwen3-ASR-0.6B
+modelscope download --model Qwen/Qwen3-ForcedAligner-0.6B --local_dir ./Qwen3-ForcedAligner-0.6B
+# Download through Hugging Face
+pip install -U "huggingface_hub[cli]"
+huggingface-cli download Qwen/Qwen3-ASR-1.7B --local-dir ./Qwen3-ASR-1.7B
+huggingface-cli download Qwen/Qwen3-ASR-0.6B --local-dir ./Qwen3-ASR-0.6B
+huggingface-cli download Qwen/Qwen3-ForcedAligner-0.6B --local-dir ./Qwen3-ForcedAligner-0.6B
+```
+
+
+## Quickstart
+
+### Environment Setup
+
+The easiest way to use Qwen3-ASR is to install the `qwen-asr` Python package from PyPI. This will pull in the required runtime dependencies and allow you to load any released Qwen3-ASR model. If you’d like to simplify environment setup further, you can also use our official [Docker image](#docker). The `qwen-asr` package provides two backends: the transformers backend and the vLLM backend. For usage instructions for different backends, please refer to [Python Package Usage](#python-package-usage). We recommend using a **fresh, isolated environment** to avoid dependency conflicts with existing packages. You can create a clean Python 3.12 environment like this:
+
+```bash
+conda create -n qwen3-asr python=3.12 -y
+conda activate qwen3-asr
+```
+
+Run the following command to get the minimal installation with transformers-backend support:
+
+```bash
+pip install -U qwen-asr
+```
+
+To enable the vLLM backend for faster inference and streaming support, run:
+
+```bash
+pip install -U qwen-asr[vllm]
+```
+
+If you want to develop or modify the code locally, install from source in editable mode:
+
+```bash
+git clone https://github.com/QwenLM/Qwen3-ASR.git
+cd Qwen3-ASR
+pip install -e .
+# support vLLM backend
+# pip install -e ".[vllm]"
+```
+
+Additionally, we recommend using FlashAttention 2 to reduce GPU memory usage and accelerate inference speed, especially for long inputs and large batch sizes.
+
+```bash
+pip install -U flash-attn --no-build-isolation
+```
+
+If your machine has less than 96GB of RAM and lots of CPU cores, run:
+
+```bash
+MAX_JOBS=4 pip install -U flash-attn --no-build-isolation
+```
+
+Also, you should have hardware that is compatible with FlashAttention 2. Read more about it in the official documentation of the [FlashAttention repository](https://github.com/Dao-AILab/flash-attention). FlashAttention 2 can only be used when a model is loaded in `torch.float16` or `torch.bfloat16`.
+
+### Python Package Usage
+
+#### Quick Inference
+
+The `qwen-asr` package provides two backends: **transformers backend** and **vLLM backend**. You can pass audio inputs as a local path, a URL, base64 data, or a `(np.ndarray, sr)` tuple, and run batch inference. To quickly try Qwen3-ASR, you can use `Qwen3ASRModel.from_pretrained(...)` for the transformers backend with the following code:
+
+```python
+import torch
+from qwen_asr import Qwen3ASRModel
+
+model = Qwen3ASRModel.from_pretrained(
+    "Qwen/Qwen3-ASR-1.7B",
+    dtype=torch.bfloat16,
+    device_map="cuda:0",
+    # attn_implementation="flash_attention_2",
+    max_inference_batch_size=32, # Batch size limit for inference. -1 means unlimited. Smaller values can help avoid OOM.
+    max_new_tokens=256, # Maximum number of tokens to generate. Set a larger value for long audio input.
+)
+
+results = model.transcribe(
+    audio="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav",
+    language=None, # set "English" to force the language
+)
+
+print(results[0].language)
+print(results[0].text)
+```
+
+If you want to return timestamps, pass `forced_aligner` and its init kwargs. Here is an example of batch inference with timestamps output:
+
+```python
+import torch
+from qwen_asr import Qwen3ASRModel
+
+model = Qwen3ASRModel.from_pretrained(
+    "Qwen/Qwen3-ASR-1.7B",
+    dtype=torch.bfloat16,
+    device_map="cuda:0",
+    # attn_implementation="flash_attention_2",
+    max_inference_batch_size=32, # Batch size limit for inference. -1 means unlimited. Smaller values can help avoid OOM.
+    max_new_tokens=256, # Maximum number of tokens to generate. Set a larger value for long audio input.
+    forced_aligner="Qwen/Qwen3-ForcedAligner-0.6B",
+    forced_aligner_kwargs=dict(
+        dtype=torch.bfloat16,
+        device_map="cuda:0",
+        # attn_implementation="flash_attention_2",
+    ),
+)
+
+results = model.transcribe(
+    audio=[
+      "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_zh.wav",
+      "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav",
+    ],
+    language=["Chinese", "English"], # can also be set to None for automatic language detection
+    return_time_stamps=True,
+)
+
+for r in results:
+    print(r.language, r.text, r.time_stamps[0])
+```
+
+For more detailed usage examples, please refer to the [example code](https://github.com/QwenLM/Qwen3-ASR/blob/main/examples/example_qwen3_asr_transformers.py) for the transformers backend.
+
+#### vLLM Backend
+
+If you want the fastest inference speed with Qwen3-ASR, we strongly recommend using the vLLM backend by initializing the model with `Qwen3ASRModel.LLM(...)`. Example code is provided below. Note that you must install it via `pip install -U qwen-asr[vllm]`. If you want the model to output timestamps, it’s best to install FlashAttention via `pip install -U flash-attn --no-build-isolation` to speed up inference for the forced aligner model. Remember to wrap your code under `if __name__ == '__main__':` to avoid the `spawn` error described in [vLLM Troubleshooting](https://docs.vllm.ai/en/latest/usage/troubleshooting/#python-multiprocessing).
+
+```python
+import torch
+from qwen_asr import Qwen3ASRModel
+
+if __name__ == '__main__':
+    model = Qwen3ASRModel.LLM(
+        model="Qwen/Qwen3-ASR-1.7B",
+        gpu_memory_utilization=0.7,
+        max_inference_batch_size=128, # Batch size limit for inference. -1 means unlimited. Smaller values can help avoid OOM.
+        max_new_tokens=4096, # Maximum number of tokens to generate. Set a larger value for long audio input.
+        forced_aligner="Qwen/Qwen3-ForcedAligner-0.6B",
+        forced_aligner_kwargs=dict(
+            dtype=torch.bfloat16,
+            device_map="cuda:0",
+            # attn_implementation="flash_attention_2",
+        ),
+    )
+
+    results = model.transcribe(
+        audio=[
+        "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_zh.wav",
+        "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav",
+        ],
+        language=["Chinese", "English"], # can also be set to None for automatic language detection
+        return_time_stamps=True,
+    )
+
+    for r in results:
+        print(r.language, r.text, r.time_stamps[0])
+```
+
+For more detailed usage examples, please refer to the [example code](https://github.com/QwenLM/Qwen3-ASR/blob/main/examples/example_qwen3_asr_vllm.py) for the vLLM backend. In addition, you can start a vLLM server via the `qwen-asr-serve` command, which is a wrapper around `vllm serve`. You can pass any arguments supported by `vllm serve`, for example:
+
+```bash
+qwen-asr-serve Qwen/Qwen3-ASR-1.7B --gpu-memory-utilization 0.8 --host 0.0.0.0 --port 8000
+```
+
+And send requests to the server via:
+
+```python
+import requests
+
+url = "http://localhost:8000/v1/chat/completions"
+headers = {"Content-Type": "application/json"}
+
+data = {
+    "messages": [
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "audio_url",
+                    "audio_url": {
+                        "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"
+                    },
+                }
+            ],
+        }
+    ]
+}
+
+response = requests.post(url, headers=headers, json=data, timeout=300)
+response.raise_for_status()
+content = response.json()['choices'][0]['message']['content']
+print(content)
+
+# parse ASR output if you want
+from qwen_asr import parse_asr_output
+language, text = parse_asr_output(content)
+print(language)
+print(text)
+```
+
+#### Streaming Inference
+
+Qwen3-ASR fully supports streaming inference. Currently, streaming inference is only available with the vLLM backend. Note that streaming inference does not support batch inference or returning timestamps. Please refer to the [example code](https://github.com/QwenLM/Qwen3-ASR/blob/main/examples/example_qwen3_asr_vllm_streaming.py) for details. You can also launch a streaming web demo through the [guide](#streaming-demo) to experience Qwen3-ASR’s streaming transcription capabilities. 
+
+#### ForcedAligner Usage
+
+`Qwen3-ForcedAligner-0.6B` can align text–speech pairs and return word or character level timestamps. Here is an example of using the forced aligner directly:
+
+```python
+import torch
+from qwen_asr import Qwen3ForcedAligner
+
+model = Qwen3ForcedAligner.from_pretrained(
+    "Qwen/Qwen3-ForcedAligner-0.6B",
+    dtype=torch.bfloat16,
+    device_map="cuda:0",
+    # attn_implementation="flash_attention_2",
+)
+
+results = model.align(
+    audio="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_zh.wav",
+    text="甚至出现交易几乎停滞的情况。",
+    language="Chinese",
+)
+
+print(results[0])
+print(results[0][0].text, results[0][0].start_time, results[0][0].end_time)
+```
+
+In addition, the forced aligner supports local paths / URLs / base64 data / `(np.ndarray, sr)` inputs and batch inference. Please refer to the [example code](https://github.com/QwenLM/Qwen3-ASR/blob/main/examples/example_qwen3_forced_aligner.py) for details.
+
+### DashScope API Usage
+
+To further explore Qwen3-ASR, we encourage you to try our DashScope API for a faster and more efficient experience. For detailed API information and documentation, please refer to the following:
+
+| API Description | API Documentation (Mainland China) | API Documentation (International) |
+|------------------|-----------------------------------|------------------------------------|
+| Real-time API for Qwen3-ASR. | [https://help.aliyun.com/zh/model-studio/qwen-real-time-speech-recognition](https://help.aliyun.com/zh/model-studio/qwen-real-time-speech-recognition) | [https://www.alibabacloud.com/help/en/model-studio/qwen-real-time-speech-recognition](https://www.alibabacloud.com/help/en/model-studio/qwen-real-time-speech-recognition) |
+| FileTrans API for Qwen3-ASR. | [https://help.aliyun.com/zh/model-studio/qwen-speech-recognition](https://help.aliyun.com/zh/model-studio/qwen-speech-recognition) | [https://www.alibabacloud.com/help/en/model-studio/qwen-speech-recognition](https://www.alibabacloud.com/help/en/model-studio/qwen-speech-recognition) |
+
+
+## Launch Local Web UI Demo
+
+### Gradio Demo
+
+To launch the Qwen3-ASR web UI gradio demo, install the `qwen-asr` package and run `qwen-asr-demo`. Use the command below for help:
+
+```bash
+qwen-asr-demo --help
+```
+
+To launch the demo, you can use the following commands:
+
+```bash
+# Transformers backend
+qwen-asr-demo \
+  --asr-checkpoint Qwen/Qwen3-ASR-1.7B \
+  --backend transformers \
+  --cuda-visible-devices 0 \
+  --ip 0.0.0.0 --port 8000
+
+# Transformers backend + Forced Aligner (enable timestamps)
+qwen-asr-demo \
+  --asr-checkpoint Qwen/Qwen3-ASR-1.7B \
+  --aligner-checkpoint Qwen/Qwen3-ForcedAligner-0.6B \
+  --backend transformers \
+  --cuda-visible-devices 0 \
+  --backend-kwargs '{"device_map":"cuda:0","dtype":"bfloat16","max_inference_batch_size":8,"max_new_tokens":256}' \
+  --aligner-kwargs '{"device_map":"cuda:0","dtype":"bfloat16"}' \
+  --ip 0.0.0.0 --port 8000
+
+# vLLM backend + Forced Aligner (enable timestamps)
+qwen-asr-demo \
+  --asr-checkpoint Qwen/Qwen3-ASR-1.7B \
+  --aligner-checkpoint Qwen/Qwen3-ForcedAligner-0.6B \
+  --backend vllm \
+  --cuda-visible-devices 0 \
+  --backend-kwargs '{"gpu_memory_utilization":0.7,"max_inference_batch_size":8,"max_new_tokens":2048}' \
+  --aligner-kwargs '{"device_map":"cuda:0","dtype":"bfloat16"}' \
+  --ip 0.0.0.0 --port 8000
+```
+
+Then open `http://<your-ip>:8000`, or access it via port forwarding in tools like VS Code.
+
+#### Backend Notes
+
+This demo supports two backends: transformers and vLLM. All backend-specific initialization parameters should be passed via `--backend-kwargs` as a JSON dict. If not provided, the demo will use sensible defaults.
+
+```bash
+# Example: override transformers init args without flash attention
+--backend-kwargs '{"device_map":"cuda:0","dtype":"bfloat16"}'
+
+# Example: override vLLM init args with 65% GPU memory
+--backend-kwargs '{"gpu_memory_utilization":0.65}'
+```
+
+#### CUDA Device Notes
+
+Because vLLM does not follow `cuda:0` style device selection, this demo selects GPUs by setting `CUDA_VISIBLE_DEVICES` via `--cuda-visible-devices`.
+
+```bash
+# Use GPU 0
+--cuda-visible-devices 0
+
+# Use GPU 1
+--cuda-visible-devices 1
+```
+
+#### Timestamps Notes
+
+Timestamps are only available when `--aligner-checkpoint` is provided. If you launch the demo without a forced aligner, the timestamps UI will be hidden automatically.
+
+```bash
+# No forced aligner
+qwen-asr-demo --asr-checkpoint Qwen/Qwen3-ASR-1.7B
+
+# With forced aligner
+qwen-asr-demo \
+  --asr-checkpoint Qwen/Qwen3-ASR-1.7B \
+  --aligner-checkpoint Qwen/Qwen3-ForcedAligner-0.6B
+```
+
+#### HTTPS Notes
+
+To avoid browser microphone permission issues after deploying the server, it is recommended/required to run the gradio service over HTTPS (especially when accessed remotely or behind modern browsers/gateways). Use `--ssl-certfile` and `--ssl-keyfile` to enable HTTPS. First, generate a private key and a self-signed certificate (valid for 365 days):
+
+```bash
+openssl req -x509 -newkey rsa:2048 \
+  -keyout key.pem -out cert.pem \
+  -days 365 -nodes \
+  -subj "/CN=localhost"
+```
+
+Then run the demo with HTTPS:
+
+```bash
+qwen-asr-demo \
+  --asr-checkpoint Qwen/Qwen3-ASR-1.7B \
+  --backend transformers \
+  --cuda-visible-devices 0 \
+  --ip 0.0.0.0 --port 8000 \
+  --ssl-certfile cert.pem \
+  --ssl-keyfile key.pem \
+  --no-ssl-verify
+```
+
+Then open `https://<your-ip>:8000` to use it. If your browser shows a warning, that’s expected for self-signed certificates. For production, use a real certificate.
+
+### Streaming Demo
+
+To experience Qwen3-ASR’s streaming transcription capability in a web UI, we provide a minimal Flask-based streaming demo. The demo captures microphone audio in the browser, resamples it to 16,000 Hz, and continuously pushes PCM chunks to the model. Run the demo with the following command:
+
+```bash
+qwen-asr-demo-streaming \
+  --asr-model-path Qwen/Qwen3-ASR-1.7B \
+  --host 0.0.0.0 \
+  --port 8000 \
+  --gpu-memory-utilization 0.9
+```
+
+Then open `http://<your-ip>:8000`, or access it via port forwarding in tools like VS Code.
+
+## Deployment with vLLM
+
+vLLM officially provides day-0 model support for Qwen3-ASR for efficient inference. 
+
+### Installation
+You can run Qwen3-ASR with vLLM nightly wheel or docker image. To install the nightly version of vLLM, we recommend using `uv` as the environment manager
+```bash
+uv venv
+source .venv/bin/activate
+uv pip install -U vllm --pre \
+    --extra-index-url https://wheels.vllm.ai/nightly/cu129 \
+    --extra-index-url https://download.pytorch.org/whl/cu129 \
+    --index-strategy unsafe-best-match
+uv pip install "vllm[audio]" # For additional audio dependencies
+```
+
+### Online Serving
+You can easily deploy Qwen3-ASR with vLLM by running the following command
+```bash
+vllm serve Qwen/Qwen3-ASR-1.7B
+```
+After the model server is successfully deployed, you can interact with it in multiple ways.
+
+#### Using OpenAI SDK
+```python
+import base64
+import httpx
+from openai import OpenAI
+
+# Initialize client
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Create multimodal chat completion request
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-ASR-1.7B",
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "audio_url",
+                    "audio_url": {
+                        {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"}
+                    }
+                }
+            ]
+        }
+    ],
+)
+
+print(response.choices[0].message.content)
+```
+This model is also supported on vLLM with OpenAI transcription API.
+```python
+import httpx
+from openai import OpenAI
+
+# Initialize client
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+audio_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"
+audio_file = httpx.get(audio_url).content
+
+transcription = client.audio.transcriptions.create(
+    model="Qwen/Qwen3-ASR-1.7B",
+    file=audio_file,
+)
+
+print(transcription.text)
+```
+
+#### Using cURL
+```bash
+curl http://localhost:8000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+    "messages": [
+    {"role": "user", "content": [
+        {"type": "audio_url", "audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"}}
+    ]}
+    ]
+    }'
+```
+
+### Offline Inference
+See the following example on using vLLM to run offline inference with Qwen3-ASR
+```python
+from vllm import LLM, SamplingParams
+from vllm.assets.audio import AudioAsset
+import base64
+import requests
+
+# Initialize the LLM
+llm = LLM(
+    model="Qwen/Qwen3-ASR-1.7B"
+)
+
+# Load audio
+audio_asset = AudioAsset("winning_call")
+
+# Create conversation with audio content
+conversation = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "audio_url",
+                "audio_url": {"url": audio_asset.url}
+            }
+        ]
+    }
+]
+
+sampling_params = SamplingParams(temperature=0.01, max_tokens=256)
+
+# Run inference using .chat()
+outputs = llm.chat(conversation, sampling_params=sampling_params)
+print(outputs[0].outputs[0].text)
+```
+
+
+## Docker
+
+To make it easier to use our `qwen-asr` Python package, we provide a pre-built Docker image: [qwenllm/qwen3-asr](https://hub.docker.com/r/qwenllm/qwen3-asr). You only need to install the GPU driver and download the model files to run the code. Please follow the [NVIDIA Container Toolkit installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) to ensure Docker can access your GPU. If you are in Mainland China and have trouble reaching Docker Hub, you may use a registry mirror to accelerate image pulls.
+
+First, pull the image and start a container:
+
+```bash
+LOCAL_WORKDIR=/path/to/your/workspace
+HOST_PORT=8000
+CONTAINER_PORT=80
+docker run --gpus all --name qwen3-asr \
+    -v /var/run/docker.sock:/var/run/docker.sock -p $HOST_PORT:$CONTAINER_PORT \
+    --mount type=bind,source=$LOCAL_WORKDIR,target=/data/shared/Qwen3-ASR \
+    --shm-size=4gb \
+    -it qwenllm/qwen3-asr:latest
+```
+
+After running the command, you will enter the container’s bash shell. Your local workspace (**replace** `/path/to/your/workspace` **with the actual path**) will be mounted inside the container at `/data/shared/Qwen3-ASR`. Port `8000` on the host is mapped to port `80` in the container, so you can access services running in the container via `http://<host-ip>:8000`. Note that services inside the container must bind to `0.0.0.0` (not `127.0.0.1`) for port forwarding to work.
+
+If you exit the container, you can start it again and re-enter it with:
+
+```bash
+docker start qwen3-asr
+docker exec -it qwen3-asr bash
+```
+
+To remove the container completely, run:
+
+```bash
+docker rm -f qwen3-asr
+```
+
+
+## Evaluation
+
+During evaluation, we ran inference for all models with `dtype=torch.bfloat16` and set `max_new_tokens=1024` using vLLM. Greedy search was used for all decoding, and none of the tests specified a language parameter. The detailed evaluation results are shown below.
+
+<details>
+<summary>ASR Benchmarks on Public Datasets (WER ↓)</summary>
+
+<table>
+  <thead>
+    <tr>
+      <th colspan="2" style="text-align: left;"></th>
+      <th style="text-align: center;">GPT-4o<br>-Transcribe</th>
+      <th style="text-align: center;">Gemini-2.5<br>-Pro</th>
+      <th style="text-align: center;">Doubao-ASR</th>
+      <th style="text-align: center;">Whisper<br>-large-v3</th>
+      <th style="text-align: center;">Fun-ASR<br>-MLT-Nano</th>
+      <th style="text-align: center;">Qwen3-ASR<br>-0.6B</th>
+      <th style="text-align: center;">Qwen3-ASR<br>-1.7B</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td colspan="9" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">English (en)</td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">Librispeech<br>clean | other</td>
+      <td style="text-align: center;"><strong>1.39</strong> | 3.75</td>
+      <td style="text-align: center;">2.89 | 3.56</td>
+      <td style="text-align: center;">2.78 | 5.70</td>
+      <td style="text-align: center;">1.51 | 3.97</td>
+      <td style="text-align: center;">1.68 | 4.03</td>
+      <td style="text-align: center;">2.11 | 4.55</td>
+      <td style="text-align: center;">1.63 | <strong>3.38</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">GigaSpeech</td>
+      <td style="text-align: center;">25.50</td>
+      <td style="text-align: center;">9.37</td>
+      <td style="text-align: center;">9.55</td>
+      <td style="text-align: center;">9.76</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">8.88</td>
+      <td style="text-align: center;"><strong>8.45</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">CV-en</td>
+      <td style="text-align: center;">9.08</td>
+      <td style="text-align: center;">14.49</td>
+      <td style="text-align: center;">13.78</td>
+      <td style="text-align: center;">9.90</td>
+      <td style="text-align: center;">9.90</td>
+      <td style="text-align: center;">9.92</td>
+      <td style="text-align: center;"><strong>7.39</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">Fleurs-en</td>
+      <td style="text-align: center;"><strong>2.40</strong></td>
+      <td style="text-align: center;">2.94</td>
+      <td style="text-align: center;">6.31</td>
+      <td style="text-align: center;">4.08</td>
+      <td style="text-align: center;">5.49</td>
+      <td style="text-align: center;">4.39</td>
+      <td style="text-align: center;">3.35</td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">MLS-en</td>
+      <td style="text-align: center;">5.12</td>
+      <td style="text-align: center;"><strong>3.68</strong></td>
+      <td style="text-align: center;">7.09</td>
+      <td style="text-align: center;">4.87</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">6.00</td>
+      <td style="text-align: center;">4.58</td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">Tedlium</td>
+      <td style="text-align: center;">7.69</td>
+      <td style="text-align: center;">6.15</td>
+      <td style="text-align: center;">4.91</td>
+      <td style="text-align: center;">6.84</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>3.85<strong></td>
+      <td style="text-align: center;"><strong>4.50</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">VoxPopuli</td>
+      <td style="text-align: center;">10.29</td>
+      <td style="text-align: center;">11.36</td>
+      <td style="text-align: center;">12.12</td>
+      <td style="text-align: center;">12.05</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>9.96<strong></td>
+      <td style="text-align: center;"><strong>9.15</strong></td>
+    </tr>
+    <tr>
+      <td colspan="9" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">Chinese (zh)</td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">WenetSpeech<br>net | meeting</td>
+      <td style="text-align: center;">15.30 | 32.27</td>
+      <td style="text-align: center;">14.43 | 13.47</td>
+      <td style="text-align: center;">N/A</td>
+      <td style="text-align: center;">9.86 | 19.11</td>
+      <td style="text-align: center;">6.35 | -</td>
+      <td style="text-align: center;">5.97 | 6.88</td>
+      <td style="text-align: center;"><strong>4.97</strong> | <strong>5.88</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">AISHELL-2-test</td>
+      <td style="text-align: center;">4.24</td>
+      <td style="text-align: center;">11.62</td>
+      <td style="text-align: center;">2.85</td>
+      <td style="text-align: center;">5.06</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">3.15</td>
+      <td style="text-align: center;"><strong>2.71</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">SpeechIO</td>
+      <td style="text-align: center;">12.86</td>
+      <td style="text-align: center;">5.30</td>
+      <td style="text-align: center;">2.93</td>
+      <td style="text-align: center;">7.56</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">3.44</td>
+      <td style="text-align: center;"><strong>2.88</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">Fleurs-zh</td>
+      <td style="text-align: center;">2.44</td>
+      <td style="text-align: center;">2.71</td>
+      <td style="text-align: center;">2.69</td>
+      <td style="text-align: center;">4.09</td>
+      <td style="text-align: center;">3.51</td>
+      <td style="text-align: center;">2.88</td>
+      <td style="text-align: center;"><strong>2.41</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">CV-zh</td>
+      <td style="text-align: center;">6.32</td>
+      <td style="text-align: center;">7.70</td>
+      <td style="text-align: center;">5.95</td>
+      <td style="text-align: center;">12.91</td>
+      <td style="text-align: center;">6.20</td>
+      <td style="text-align: center;">6.89</td>
+      <td style="text-align: center;"><strong>5.35</strong></td>
+    </tr>
+    <tr>
+      <td colspan="9" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">Chinese Dialect</td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">KeSpeech</td>
+      <td style="text-align: center;">26.87</td>
+      <td style="text-align: center;">24.71</td>
+      <td style="text-align: center;">5.27</td>
+      <td style="text-align: center;">28.79</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">7.08</td>
+      <td style="text-align: center;"><strong>5.10</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">Fleurs-yue</td>
+      <td style="text-align: center;">4.98</td>
+      <td style="text-align: center;">9.43</td>
+      <td style="text-align: center;">4.98</td>
+      <td style="text-align: center;">9.18</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">5.79</td>
+      <td style="text-align: center;"><strong>3.98</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">CV-yue</td>
+      <td style="text-align: center;">11.36</td>
+      <td style="text-align: center;">18.76</td>
+      <td style="text-align: center;">13.20</td>
+      <td style="text-align: center;">16.23</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">9.50</td>
+      <td style="text-align: center;"><strong>7.57</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">CV-zh-tw</td>
+      <td style="text-align: center;">6.32</td>
+      <td style="text-align: center;">7.31</td>
+      <td style="text-align: center;">4.06</td>
+      <td style="text-align: center;">7.84</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">5.59</td>
+      <td style="text-align: center;"><strong>3.77</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">WenetSpeech-Yue<br>short | long</td>
+      <td style="text-align: center;">15.62 | 25.29</td>
+      <td style="text-align: center;">25.19 | 11.23</td>
+      <td style="text-align: center;">9.74 | 11.40</td>
+      <td style="text-align: center;">32.26 | 46.64</td>
+      <td style="text-align: center;">- | -</td>
+      <td style="text-align: center;">7.54 | 9.92</td>
+      <td style="text-align: center;"><strong>5.82</strong> | <strong>8.85</strong></td>
+    </tr>
+    <tr>
+      <td colspan="2" style="text-align: left;">WenetSpeech-Chuan<br>easy | hard</td>
+      <td style="text-align: center;">34.81 | 53.98</td>
+      <td style="text-align: center;">43.79 | 67.30</td>
+      <td style="text-align: center;"><strong>11.40<strong> | <strong>20.20</strong></td>
+      <td style="text-align: center;">14.35 | 26.80</td>
+      <td style="text-align: center;">- | -</td>
+      <td style="text-align: center;">13.92 | 24.45</td>
+      <td style="text-align: center;">11.99 | 21.63</td>
+    </tr>
+  </tbody>
+</table>
+
+</details>
+
+<details>
+<summary>ASR Benchmarks on Internal Datasets (WER ↓)</summary>
+
+<table>
+  <thead>
+    <tr>
+      <th style="text-align: left;"></th>
+      <th style="text-align: center;">GPT-4o<br>-Transcribe</th>
+      <th style="text-align: center;">Gemini-2.5<br>-Pro</th>
+      <th style="text-align: center;">Doubao-ASR</th>
+      <th style="text-align: center;">Whisper<br>-large-v3</th>
+      <th style="text-align: center;">Fun-ASR<br>-MLT-Nano</th>
+      <th style="text-align: center;">Qwen3-ASR<br>-0.6B</th>
+      <th style="text-align: center;">Qwen3-ASR<br>-1.7B</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td colspan="8" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">Accented English</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Dialog-Accented English</td>
+      <td style="text-align: center;">28.56</td>
+      <td style="text-align: center;">23.85</td>
+      <td style="text-align: center;">20.41</td>
+      <td style="text-align: center;">21.30</td>
+      <td style="text-align: center;">19.96</td>
+      <td style="text-align: center;"><strong>16.62<strong></td>
+      <td style="text-align: center;"><strong>16.07</strong></td>
+    </tr>
+    <tr>
+      <td colspan="8" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">Chinese Mandarin</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Elders&Kids</td>
+      <td style="text-align: center;">14.27</td>
+      <td style="text-align: center;">36.93</td>
+      <td style="text-align: center;">4.17</td>
+      <td style="text-align: center;">10.61</td>
+      <td style="text-align: center;">4.54</td>
+      <td style="text-align: center;">4.48</td>
+      <td style="text-align: center;"><strong>3.81</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">ExtremeNoise</td>
+      <td style="text-align: center;">36.11</td>
+      <td style="text-align: center;">29.06</td>
+      <td style="text-align: center;">17.04</td>
+      <td style="text-align: center;">63.17</td>
+      <td style="text-align: center;">36.55</td>
+      <td style="text-align: center;">17.88</td>
+      <td style="text-align: center;"><strong>16.17</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">TongueTwister</td>
+      <td style="text-align: center;">20.87</td>
+      <td style="text-align: center;">4.97</td>
+      <td style="text-align: center;">3.47</td>
+      <td style="text-align: center;">16.63</td>
+      <td style="text-align: center;">9.02</td>
+      <td style="text-align: center;">4.06</td>
+      <td style="text-align: center;"><strong>2.44</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Dialog-Mandarin</td>
+      <td style="text-align: center;">20.73</td>
+      <td style="text-align: center;">12.50</td>
+      <td style="text-align: center;">6.61</td>
+      <td style="text-align: center;">14.01</td>
+      <td style="text-align: center;">7.32</td>
+      <td style="text-align: center;">7.06</td>
+      <td style="text-align: center;"><strong>6.54</strong></td>
+    </tr>
+    <tr>
+      <td colspan="8" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">Chinese Dialect</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Dialog-Cantonese</td>
+      <td style="text-align: center;">16.05</td>
+      <td style="text-align: center;">14.98</td>
+      <td style="text-align: center;">7.56</td>
+      <td style="text-align: center;">31.04</td>
+      <td style="text-align: center;">5.85</td>
+      <td style="text-align: center;"><strong>4.80<strong></td>
+      <td style="text-align: center;"><strong>4.12</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Dialog-Chinese Dialects</td>
+      <td style="text-align: center;">45.37</td>
+      <td style="text-align: center;">47.70</td>
+      <td style="text-align: center;">19.85</td>
+      <td style="text-align: center;">44.55</td>
+      <td style="text-align: center;">19.41</td>
+      <td style="text-align: center;"><strong>18.24<strong></td>
+      <td style="text-align: center;"><strong>15.94</strong></td>
+    </tr>
+  </tbody>
+</table>
+<p><strong>Dialect coverage:</strong> Results for <em>Dialog-Accented English</em> are averaged over 16 accents, and results for <em>Dialog-Chinese Dialects</em> are averaged over 22 Chinese dialects.</p>
+
+</details>
+
+<details>
+<summary>Multilingual ASR Benchmarks (WER ↓)</summary>
+
+<table>
+  <thead>
+    <tr>
+      <th style="text-align: left;"></th>
+      <th style="text-align: center;">GLM-ASR<br>-Nano-2512</th>
+      <th style="text-align: center;">Whisper<br>-large-v3</th>
+      <th style="text-align: center;">Fun-ASR<br>-MLT-Nano</th>
+      <th style="text-align: center;">Qwen3-ASR<br>-0.6B</th>
+      <th style="text-align: center;">Qwen3-ASR<br>-1.7B</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td colspan="6" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">Open-sourced Benchmarks</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">MLS</td>
+      <td style="text-align: center;">13.32</td>
+      <td style="text-align: center;">8.62</td>
+      <td style="text-align: center;">28.70</td>
+      <td style="text-align: center;">13.19</td>
+      <td style="text-align: center;"><strong>8.55</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">CommonVoice</td>
+      <td style="text-align: center;">19.40</td>
+      <td style="text-align: center;">10.77</td>
+      <td style="text-align: center;">17.25</td>
+      <td style="text-align: center;">12.75</td>
+      <td style="text-align: center;"><strong>9.18</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">MLC-SLM</td>
+      <td style="text-align: center;">34.93</td>
+      <td style="text-align: center;">15.68</td>
+      <td style="text-align: center;">29.94</td>
+      <td style="text-align: center;">15.84</td>
+      <td style="text-align: center;"><strong>12.74</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Fleurs</td>
+      <td style="text-align: center;">16.08</td>
+      <td style="text-align: center;">5.27</td>
+      <td style="text-align: center;">10.03</td>
+      <td style="text-align: center;">7.57</td>
+      <td style="text-align: center;"><strong>4.90</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Fleurs<sup>†</sup></td>
+      <td style="text-align: center;">20.05</td>
+      <td style="text-align: center;">6.85</td>
+      <td style="text-align: center;">31.89</td>
+      <td style="text-align: center;">10.37</td>
+      <td style="text-align: center;"><strong>6.62</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Fleurs<sup>††</sup></td>
+      <td style="text-align: center;">24.83</td>
+      <td style="text-align: center;"><strong>8.16</strong></td>
+      <td style="text-align: center;">47.84</td>
+      <td style="text-align: center;">21.80</td>
+      <td style="text-align: center;">12.60</td>
+    </tr>
+    <tr>
+      <td colspan="6" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">Qwen-ASR Internal Benchmarks</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">News-Multilingual</td>
+      <td style="text-align: center;">49.40</td>
+      <td style="text-align: center;">14.80</td>
+      <td style="text-align: center;">65.07</td>
+      <td style="text-align: center;">17.39</td>
+      <td style="text-align: center;"><strong>12.80</strong></td>
+    </tr>
+  </tbody>
+</table>
+<p><strong>Language coverage:</strong> <em>MLS</em> includes 8 languages: {da, de, en, es, fr, it, pl, pt}.<br><em>CommonVoice</em> includes 13 languages: {en, zh, yue, zh_TW, ar, de, es, fr, it, ja, ko, pt, ru}.<br><em>MLC-SLM</em> includes 11 languages: {en, fr, de, it, pt, es, ja, ko, ru, th, vi}.<br><em>Fleurs</em> includes 12 languages: {en, zh, yue, ar, de, es, fr, it, ja, ko, pt, ru }.<br><em>Fleurs<sup>†</sup></em> includes 8 additional languages beyond Fleurs: {hi, id, ms, nl, pl, th, tr, vi}.<br><em>Fleurs<sup>††</sup></em> includes 10 additional languages beyond Fleurs<sup>†</sup>: {cs, da, el, fa, fi, fil, hu, mk, ro, sv}.<br><em>News-Multilingual</em> includes 15 languages: {ar, de, es, fr, hi, id, it, ja, ko, nl, pl, pt, ru, th, vi}.</p>
+
+</details>
+
+<details>
+<summary>Language Identification Accuracy (%) ↑</summary>
+
+<table>
+  <thead>
+    <tr>
+      <th style="text-align: left;"></th>
+      <th style="text-align: center;">Whisper-large-v3</th>
+      <th style="text-align: center;">Qwen3-ASR-0.6B</th>
+      <th style="text-align: center;">Qwen3-ASR-1.7B</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style="text-align: left;">MLS</td>
+      <td style="text-align: center;"><strong>99.9</strong></td>
+      <td style="text-align: center;">99.3</td>
+      <td style="text-align: center;"><strong>99.9</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">CommonVoice</td>
+      <td style="text-align: center;">92.7</td>
+      <td style="text-align: center;"><strong>98.2<strong></td>
+      <td style="text-align: center;"><strong>98.7</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">MLC-SLM</td>
+      <td style="text-align: center;">89.2</td>
+      <td style="text-align: center;"><strong>92.7<strong></td>
+      <td style="text-align: center;"><strong>94.1</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Fleurs</td>
+      <td style="text-align: center;">94.6</td>
+      <td style="text-align: center;"><strong>97.1<strong></td>
+      <td style="text-align: center;"><strong>98.7</strong></td>
+    </tr>
+    <tr style="border-top: 1px solid #ddd;">
+      <td style="text-align: left;"><em>Avg.</em></td>
+      <td style="text-align: center;">94.1</td>
+      <td style="text-align: center;"><strong>96.8<strong></td>
+      <td style="text-align: center;"><strong>97.9</strong></td>
+    </tr>
+  </tbody>
+</table>
+<p><strong>Language coverage:</strong> The language sets follow Multilingual ASR Benchmarks. Here, Fleurs corresponds to Fleurs<sup>††</sup> in Multilingual ASR Benchmarks and covers 30 languages.</p>
+
+</details>
+
+<details>
+<summary>Singing Voice & Song Transcription (WER ↓)</summary>
+
+<table>
+  <thead>
+    <tr>
+      <th style="text-align: left;"></th>
+      <th style="text-align: center;">GPT-4o<br>-Transcribe</th>
+      <th style="text-align: center;">Gemini-2.5<br>-Pro</th>
+      <th style="text-align: center;">Doubao-ASR<br>-1.0</th>
+      <th style="text-align: center;">Whisper<br>-large-v3</th>
+      <th style="text-align: center;">Fun-ASR-MLT<br>-Nano</th>
+      <th style="text-align: center;">Qwen3-ASR<br>-1.7B</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td colspan="7" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">Singing</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">M4Singer</td>
+      <td style="text-align: center;">16.77</td>
+      <td style="text-align: center;">20.88</td>
+      <td style="text-align: center;">7.88</td>
+      <td style="text-align: center;">13.58</td>
+      <td style="text-align: center;">7.29</td>
+      <td style="text-align: center;"><strong>5.98</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">MIR-1k-vocal</td>
+      <td style="text-align: center;">11.87</td>
+      <td style="text-align: center;">9.85</td>
+      <td style="text-align: center;">6.56</td>
+      <td style="text-align: center;">11.71</td>
+      <td style="text-align: center;">8.17</td>
+      <td style="text-align: center;"><strong>6.25</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Opencpop</td>
+      <td style="text-align: center;">7.93</td>
+      <td style="text-align: center;">6.49</td>
+      <td style="text-align: center;">3.80</td>
+      <td style="text-align: center;">9.52</td>
+      <td style="text-align: center;"><strong>2.98</strong></td>
+      <td style="text-align: center;">3.08</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Popcs</td>
+      <td style="text-align: center;">32.84</td>
+      <td style="text-align: center;">15.13</td>
+      <td style="text-align: center;">8.97</td>
+      <td style="text-align: center;">13.77</td>
+      <td style="text-align: center;">9.42</td>
+      <td style="text-align: center;"><strong>8.52</strong></td>
+    </tr>
+    <tr>
+      <td colspan="7" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">Songs with BGM</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">EntireSongs-en</td>
+      <td style="text-align: center;">30.71</td>
+      <td style="text-align: center;"><strong>12.18</strong></td>
+      <td style="text-align: center;">33.51</td>
+      <td style="text-align: center;">N/A</td>
+      <td style="text-align: center;">N/A</td>
+      <td style="text-align: center;">14.60</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">EntireSongs-zh</td>
+      <td style="text-align: center;">34.86</td>
+      <td style="text-align: center;">18.68</td>
+      <td style="text-align: center;">23.99</td>
+      <td style="text-align: center;">N/A</td>
+      <td style="text-align: center;">N/A</td>
+      <td style="text-align: center;"><strong>13.91</strong></td>
+    </tr>
+  </tbody>
+</table>
+
+</details>
+
+<details>
+<summary>ASR Inference Mode Performance (WER ↓)</summary>
+
+<table>
+  <thead>
+    <tr>
+      <th style="text-align: left;">Model</th>
+      <th style="text-align: left;">Infer. Mode</th>
+      <th style="text-align: center;">Librispeech</th>
+      <th style="text-align: center;">Fleurs-en</th>
+      <th style="text-align: center;">Fleurs-zh</th>
+      <th style="text-align: center;">Avg.</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td rowspan="2" style="text-align: left; vertical-align: middle;">Qwen3-ASR-1.7B</td>
+      <td style="text-align: left;">Offline</td>
+      <td style="text-align: center;">1.63 | 3.38</td>
+      <td style="text-align: center;">3.35</td>
+      <td style="text-align: center;">2.41</td>
+      <td style="text-align: center;">2.69</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Streaming</td>
+      <td style="text-align: center;">1.95 | 4.51</td>
+      <td style="text-align: center;">4.02</td>
+      <td style="text-align: center;">2.84</td>
+      <td style="text-align: center;">3.33</td>
+    </tr>
+    <tr style="border-top: 1px solid #ddd;">
+      <td rowspan="2" style="text-align: left; vertical-align: middle;">Qwen3-ASR-0.6B</td>
+      <td style="text-align: left;">Offline</td>
+      <td style="text-align: center;">2.11 | 4.55</td>
+      <td style="text-align: center;">4.39</td>
+      <td style="text-align: center;">2.88</td>
+      <td style="text-align: center;">3.48</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Streaming</td>
+      <td style="text-align: center;">2.54 | 6.27</td>
+      <td style="text-align: center;">5.38</td>
+      <td style="text-align: center;">3.40</td>
+      <td style="text-align: center;">4.40</td>
+    </tr>
+  </tbody>
+</table>
+
+</details>
+
+<details>
+<summary>Forced Alignment Benchmarks (AAS ms ↓)</summary>
+
+<table>
+  <thead>
+    <tr>
+      <th style="text-align: left;"></th>
+      <th style="text-align: center;">Monotonic-Aligner</th>
+      <th style="text-align: center;">NFA</th>
+      <th style="text-align: center;">WhisperX</th>
+      <th style="text-align: center;">Qwen3-ForcedAligner-0.6B</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td colspan="5" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">MFA-Labeled Raw</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Chinese</td>
+      <td style="text-align: center;">161.1</td>
+      <td style="text-align: center;">109.8</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>33.1</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">English</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">107.5</td>
+      <td style="text-align: center;">92.1</td>
+      <td style="text-align: center;"><strong>37.5</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">French</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">100.7</td>
+      <td style="text-align: center;">145.3</td>
+      <td style="text-align: center;"><strong>41.7</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">German</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">122.7</td>
+      <td style="text-align: center;">165.1</td>
+      <td style="text-align: center;"><strong>46.5</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Italian</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">142.7</td>
+      <td style="text-align: center;">155.5</td>
+      <td style="text-align: center;"><strong>75.5</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Japanese</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>42.2</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Korean</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>37.2</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Portuguese</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>38.4</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Russian</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">200.7</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>40.2</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Spanish</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">124.7</td>
+      <td style="text-align: center;">108.0</td>
+      <td style="text-align: center;"><strong>36.8</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;"><em>Avg.</em></td>
+      <td style="text-align: center;">161.1</td>
+      <td style="text-align: center;">129.8</td>
+      <td style="text-align: center;">133.2</td>
+      <td style="text-align: center;"><strong>42.9</strong></td>
+    </tr>
+    <tr>
+      <td colspan="5" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">MFA-Labeled Concat-300s</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Chinese</td>
+      <td style="text-align: center;">1742.4</td>
+      <td style="text-align: center;">235.0</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>36.5</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">English</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">226.7</td>
+      <td style="text-align: center;">227.2</td>
+      <td style="text-align: center;"><strong>58.6</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">French</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">230.6</td>
+      <td style="text-align: center;">2052.2</td>
+      <td style="text-align: center;"><strong>53.4</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">German</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">220.3</td>
+      <td style="text-align: center;">993.4</td>
+      <td style="text-align: center;"><strong>62.4</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Italian</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">290.5</td>
+      <td style="text-align: center;">5719.4</td>
+      <td style="text-align: center;"><strong>81.6</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Japanese</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>81.3</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Korean</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>42.2</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Portuguese</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>50.0</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Russian</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">283.3</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>43.0</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Spanish</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">240.2</td>
+      <td style="text-align: center;">4549.9</td>
+      <td style="text-align: center;"><strong>39.6</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Cross-lingual</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>34.2</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;"><em>Avg.</em></td>
+      <td style="text-align: center;">1742.4</td>
+      <td style="text-align: center;">246.7</td>
+      <td style="text-align: center;">2708.4</td>
+      <td style="text-align: center;"><strong>52.9</strong></td>
+    </tr>
+    <tr>
+      <td colspan="5" style="text-align: left; font-style: italic; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;">Human-Labeled</td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Raw</td>
+      <td style="text-align: center;">49.9</td>
+      <td style="text-align: center;">88.6</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>27.8</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Raw-Noisy</td>
+      <td style="text-align: center;">53.3</td>
+      <td style="text-align: center;">89.5</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>41.8</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Concat-60s</td>
+      <td style="text-align: center;">51.1</td>
+      <td style="text-align: center;">86.7</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>25.3</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Concat-300s</td>
+      <td style="text-align: center;">410.8</td>
+      <td style="text-align: center;">140.0</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>24.8</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;">Concat-Cross-lingual</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>42.5</strong></td>
+    </tr>
+    <tr>
+      <td style="text-align: left;"><em>Avg.</em></td>
+      <td style="text-align: center;">141.3</td>
+      <td style="text-align: center;">101.2</td>
+      <td style="text-align: center;">-</td>
+      <td style="text-align: center;"><strong>32.4</strong></td>
+    </tr>
+  </tbody>
+</table>
+
+</details>
+
+
+## Citation
+
+If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil: :)
+
+```BibTeX
+@article{Qwen3-ASR,
+  title={Qwen3-ASR Technical Report},
+  author={Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, Junyang Lin},
+  journal={arXiv preprint arXiv:2601.21337},
+  year={2026}
+}
+```
+
+
+<br>

Qwen3-ASR-1.7B/chat_template.json

@@ -0,0 +1 @@
+{"chat_template": "{%- set ns = namespace(system_text=\"\") -%}\n{%- for m in messages -%}\n  {%- if m.role == 'system' -%}\n    {%- if m.content is string -%}\n      {%- set ns.system_text = ns.system_text + m.content -%}\n    {%- else -%}\n      {%- for c in m.content -%}\n        {%- if c.type == 'text' and (c.text is defined) -%}\n          {%- set ns.system_text = ns.system_text + c.text -%}\n        {%- endif -%}\n      {%- endfor -%}\n    {%- endif -%}\n  {%- endif -%}\n{%- endfor -%}\n\n{%- set ns2 = namespace(audio_tokens=\"\") -%}\n{%- for m in messages -%}\n  {%- if m.content is not string -%}\n    {%- for c in m.content -%}\n      {%- if c.type == 'audio' or ('audio' in c) or ('audio_url' in c) -%}\n        {%- set ns2.audio_tokens = ns2.audio_tokens + \"<|audio_start|><|audio_pad|><|audio_end|>\" -%}\n      {%- endif -%}\n    {%- endfor -%}\n  {%- endif -%}\n{%- endfor -%}\n\n{{- '<|im_start|>system\\n' + (ns.system_text if ns.system_text is string else '') + '<|im_end|>\\n' -}}\n{{- '<|im_start|>user\\n' + ns2.audio_tokens + '<|im_end|>\\n' -}}\n{%- if add_generation_prompt -%}\n{{- '<|im_start|>assistant\\n' -}}\n{%- endif -%}"}

Qwen3-ASR-1.7B/config.json

@@ -0,0 +1,221 @@
+{
+    "architectures": [
+      "Qwen3ASRForConditionalGeneration"
+    ],
+    "model_type": "qwen3_asr",
+    "support_languages": [
+      "Chinese",
+      "English",
+      "Cantonese",
+      "Arabic",
+      "German",
+      "French",
+      "Spanish",
+      "Portuguese",
+      "Indonesian",
+      "Italian",
+      "Korean",
+      "Russian",
+      "Thai",
+      "Vietnamese",
+      "Japanese",
+      "Turkish",
+      "Hindi",
+      "Malay",
+      "Dutch",
+      "Swedish",
+      "Danish",
+      "Finnish",
+      "Polish",
+      "Czech",
+      "Filipino",
+      "Persian",
+      "Greek",
+      "Romanian",
+      "Hungarian",
+      "Macedonian"
+    ],
+    "thinker_config": {
+      "model_type": "qwen3_asr",
+      "architectures": [
+        "Qwen3ASRForConditionalGeneration"
+      ],
+      "audio_config": {
+        "_name_or_path": "",
+        "activation_dropout": 0,
+        "activation_function": "gelu",
+        "add_cross_attention": false,
+        "architectures": null,
+        "attention_dropout": 0,
+        "bad_words_ids": null,
+        "begin_suppress_tokens": null,
+        "bos_token_id": null,
+        "chunk_size_feed_forward": 0,
+        "conv_chunksize": 500,
+        "cross_attention_hidden_size": null,
+        "d_model": 1024,
+        "decoder_start_token_id": null,
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "downsample_hidden_size": 480,
+        "dropout": 0,
+        "dtype": null,
+        "early_stopping": false,
+        "encoder_attention_heads": 16,
+        "encoder_ffn_dim": 4096,
+        "encoder_layers": 24,
+        "encoder_no_repeat_ngram_size": 0,
+        "eos_token_id": null,
+        "exponential_decay_length_penalty": null,
+        "finetuning_task": null,
+        "forced_bos_token_id": null,
+        "forced_eos_token_id": null,
+        "id2label": {
+          "0": "LABEL_0",
+          "1": "LABEL_1"
+        },
+        "initializer_range": 0.02,
+        "is_decoder": false,
+        "is_encoder_decoder": false,
+        "label2id": {
+          "LABEL_0": 0,
+          "LABEL_1": 1
+        },
+        "length_penalty": 1.0,
+        "max_length": 20,
+        "max_source_positions": 1500,
+        "min_length": 0,
+        "model_type": "qwen3_asr_audio_encoder",
+        "n_window": 50,
+        "n_window_infer": 800,
+        "no_repeat_ngram_size": 0,
+        "num_beam_groups": 1,
+        "num_beams": 1,
+        "num_hidden_layers": 24,
+        "num_mel_bins": 128,
+        "num_return_sequences": 1,
+        "output_attentions": false,
+        "output_dim": 2048,
+        "output_hidden_states": false,
+        "output_scores": false,
+        "pad_token_id": null,
+        "prefix": null,
+        "problem_type": null,
+        "pruned_heads": {},
+        "remove_invalid_values": false,
+        "repetition_penalty": 1.0,
+        "return_dict": true,
+        "return_dict_in_generate": false,
+        "scale_embedding": false,
+        "sep_token_id": null,
+        "suppress_tokens": null,
+        "task_specific_params": null,
+        "temperature": 1.0,
+        "tf_legacy_loss": false,
+        "tie_encoder_decoder": false,
+        "tie_word_embeddings": true,
+        "tokenizer_class": null,
+        "top_k": 50,
+        "top_p": 1.0,
+        "torchscript": false,
+        "typical_p": 1.0,
+        "use_bfloat16": false
+      },
+      "audio_end_token_id": 151670,
+      "audio_start_token_id": 151669,
+      "audio_token_id": 151676,
+      "dtype": "bfloat16",
+      "initializer_range": 0.02,
+      "text_config": {
+        "_name_or_path": "",
+        "add_cross_attention": false,
+        "architectures": null,
+        "attention_bias": false,
+        "attention_dropout": 0.0,
+        "bad_words_ids": null,
+        "begin_suppress_tokens": null,
+        "bos_token_id": null,
+        "chunk_size_feed_forward": 0,
+        "cross_attention_hidden_size": null,
+        "decoder_start_token_id": null,
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "dtype": null,
+        "early_stopping": false,
+        "encoder_no_repeat_ngram_size": 0,
+        "eos_token_id": null,
+        "exponential_decay_length_penalty": null,
+        "finetuning_task": null,
+        "forced_bos_token_id": null,
+        "forced_eos_token_id": null,
+        "head_dim": 128,
+        "hidden_act": "silu",
+        "hidden_size": 2048,
+        "id2label": {
+          "0": "LABEL_0",
+          "1": "LABEL_1"
+        },
+        "initializer_range": 0.02,
+        "intermediate_size": 6144,
+        "is_decoder": false,
+        "is_encoder_decoder": false,
+        "label2id": {
+          "LABEL_0": 0,
+          "LABEL_1": 1
+        },
+        "length_penalty": 1.0,
+        "max_length": 20,
+        "max_position_embeddings": 65536,
+        "min_length": 0,
+        "model_type": "qwen3",
+        "no_repeat_ngram_size": 0,
+        "num_attention_heads": 16,
+        "num_beam_groups": 1,
+        "num_beams": 1,
+        "num_hidden_layers": 28,
+        "num_key_value_heads": 8,
+        "num_return_sequences": 1,
+        "output_attentions": false,
+        "output_hidden_states": false,
+        "output_scores": false,
+        "pad_token_id": null,
+        "prefix": null,
+        "problem_type": null,
+        "pruned_heads": {},
+        "remove_invalid_values": false,
+        "repetition_penalty": 1.0,
+        "return_dict": true,
+        "return_dict_in_generate": false,
+        "rms_norm_eps": 1e-06,
+        "rope_scaling": {
+          "interleaved": true,
+          "mrope_interleaved": true,
+          "mrope_section": [
+            24,
+            20,
+            20
+          ],
+          "rope_type": "default",
+          "type": "default"
+        },
+        "rope_theta": 1000000,
+        "sep_token_id": null,
+        "suppress_tokens": null,
+        "task_specific_params": null,
+        "temperature": 1.0,
+        "tf_legacy_loss": false,
+        "tie_encoder_decoder": false,
+        "tie_word_embeddings": true,
+        "tokenizer_class": null,
+        "top_k": 50,
+        "top_p": 1.0,
+        "torchscript": false,
+        "typical_p": 1.0,
+        "use_bfloat16": false,
+        "use_cache": true,
+        "vocab_size": 151936
+      }
+    },
+    "transformers_version": "4.57.6"
+  }
+  

Qwen3-ASR-1.7B/generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    151643,
+    151645
+  ],
+  "pad_token_id": 151643,
+  "do_sample": false
+}

Qwen3-ASR-1.7B/model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4cd1f1a04d90b757dc7f7dd26254e69a013b19e80efe590a83c6a3bde8608d6
+size 4220320824

Qwen3-ASR-1.7B/model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e0b9d9e09e2e0238e7ef3cc8a484ab387e91b90f1900bedf88bc92d7929ccfc
+size 478200688

Qwen3-ASR-1.7B/model.safetensors.index.json

@@ -0,0 +1,715 @@
+{
+  "metadata": {
+    "format": "pt"
+  },
+  "weight_map": {
+    "thinker.audio_tower.conv2d1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.conv2d1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.conv2d2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.conv2d2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.conv2d3.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.conv2d3.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.conv_out.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.0.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.1.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.10.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.11.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.12.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.13.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.14.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.15.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.16.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.17.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.18.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.19.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.2.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.20.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.21.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.22.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.23.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.3.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.4.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.5.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.6.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.7.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.8.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.fc1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.fc1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.fc2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.fc2.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.layers.9.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.ln_post.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.ln_post.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.proj1.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.proj1.weight": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.proj2.bias": "model-00001-of-00002.safetensors",
+    "thinker.audio_tower.proj2.weight": "model-00001-of-00002.safetensors",
+    "thinker.lm_head.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.0.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.0.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.1.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.1.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.10.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.10.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.11.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.11.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.12.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.12.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.13.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.13.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.14.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.14.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.15.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.15.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.16.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.16.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.17.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.17.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.18.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.18.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.19.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.19.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.2.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.2.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.20.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.20.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.21.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.21.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.21.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.22.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.22.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.22.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.23.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.23.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.23.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.24.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.24.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.24.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.24.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.25.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.25.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.25.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.25.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.26.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.26.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.26.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.26.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.27.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.27.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.27.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.27.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.27.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.3.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.3.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.4.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.4.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "thinker.model.layers.5.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.5.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.5.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.5.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.5.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.5.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.5.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.5.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.5.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.6.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.6.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.6.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.6.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.6.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.6.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.6.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.6.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.6.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.6.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.6.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.7.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.7.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.7.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.7.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.7.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.7.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.7.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.7.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.7.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.7.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.7.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.8.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.8.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.8.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.8.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.8.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.8.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.8.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.8.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.8.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.8.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.8.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.9.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.9.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.9.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.9.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.9.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.9.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.9.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.9.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.9.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.9.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.layers.9.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "thinker.model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}

Qwen3-ASR-1.7B/preprocessor_config.json

@@ -0,0 +1,14 @@
+{
+  "chunk_length": 30,
+  "dither": 0.0,
+  "feature_extractor_type": "WhisperFeatureExtractor",
+  "feature_size": 128,
+  "hop_length": 160,
+  "n_fft": 400,
+  "n_samples": 480000,
+  "nb_max_frames": 3000,
+  "padding_side": "right",
+  "padding_value": 0.0,
+  "processor_class": "Qwen3ASRProcessor",
+  "return_attention_mask": true
+}

Qwen3-ASR-1.7B/tokenizer_config.json

@@ -0,0 +1,549 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151669": {
+      "content": "<|audio_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151670": {
+      "content": "<|audio_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151671": {
+      "content": "<tts_pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151672": {
+      "content": "<tts_text_bos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151673": {
+      "content": "<tts_text_eod>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151674": {
+      "content": "<tts_text_bos_single>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151675": {
+      "content": "<non_speech>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151676": {
+      "content": "<|audio_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151677": {
+      "content": "<blank1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151678": {
+      "content": "<blank2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151679": {
+      "content": "<blank3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151680": {
+      "content": "<blank4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151681": {
+      "content": "<blank5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151682": {
+      "content": "<blank6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151683": {
+      "content": "<blank7>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151684": {
+      "content": "<blank8>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151685": {
+      "content": "<blank9>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151686": {
+      "content": "<blank10>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151687": {
+      "content": "<blank11>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151688": {
+      "content": "<blank12>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151689": {
+      "content": "<blank13>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151690": {
+      "content": "<blank14>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151691": {
+      "content": "<blank15>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151692": {
+      "content": "<blank16>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151693": {
+      "content": "<blank17>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151694": {
+      "content": "<blank18>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151695": {
+      "content": "<blank19>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151696": {
+      "content": "<blank20>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151697": {
+      "content": "<blank21>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151698": {
+      "content": "<blank22>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151699": {
+      "content": "<blank23>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151700": {
+      "content": "<blank24>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151701": {
+      "content": "<blank25>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151702": {
+      "content": "<blank26>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151703": {
+      "content": "<blank27>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151704": {
+      "content": "<asr_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>",
+    "<|audio_start|>",
+    "<|audio_end|>",
+    "<tts_pad>",
+    "<tts_text_bos>",
+    "<tts_text_bos_single>",
+    "<|audio_pad|>"
+  ],
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "image_token": "<|image_pad|>",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "processor_class": "Qwen3ASRProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}

audio_quality_router/best_acc_model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4539baeb97ef97b19648e79990b5966345805e2bef8760b28a8620544300a873
+size 12359420

mega-asr-merged/adapter_config.json

@@ -0,0 +1,1099 @@
+{
+  "alpha_pattern": {
+    "thinker.audio_tower.conv_out": 24,
+    "thinker.audio_tower.layers.0.fc1": 8,
+    "thinker.audio_tower.layers.0.fc2": 8,
+    "thinker.audio_tower.layers.0.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.0.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.0.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.0.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.1.fc1": 8,
+    "thinker.audio_tower.layers.1.fc2": 8,
+    "thinker.audio_tower.layers.1.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.1.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.1.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.1.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.10.fc1": 8,
+    "thinker.audio_tower.layers.10.fc2": 8,
+    "thinker.audio_tower.layers.10.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.10.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.10.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.10.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.11.fc1": 8,
+    "thinker.audio_tower.layers.11.fc2": 8,
+    "thinker.audio_tower.layers.11.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.11.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.11.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.11.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.12.fc1": 8,
+    "thinker.audio_tower.layers.12.fc2": 8,
+    "thinker.audio_tower.layers.12.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.12.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.12.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.12.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.13.fc1": 8,
+    "thinker.audio_tower.layers.13.fc2": 8,
+    "thinker.audio_tower.layers.13.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.13.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.13.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.13.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.14.fc1": 8,
+    "thinker.audio_tower.layers.14.fc2": 8,
+    "thinker.audio_tower.layers.14.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.14.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.14.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.14.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.15.fc1": 8,
+    "thinker.audio_tower.layers.15.fc2": 8,
+    "thinker.audio_tower.layers.15.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.15.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.15.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.15.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.16.fc1": 8,
+    "thinker.audio_tower.layers.16.fc2": 8,
+    "thinker.audio_tower.layers.16.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.16.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.16.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.16.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.17.fc1": 8,
+    "thinker.audio_tower.layers.17.fc2": 8,
+    "thinker.audio_tower.layers.17.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.17.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.17.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.17.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.18.fc1": 8,
+    "thinker.audio_tower.layers.18.fc2": 8,
+    "thinker.audio_tower.layers.18.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.18.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.18.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.18.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.19.fc1": 8,
+    "thinker.audio_tower.layers.19.fc2": 8,
+    "thinker.audio_tower.layers.19.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.19.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.19.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.19.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.2.fc1": 8,
+    "thinker.audio_tower.layers.2.fc2": 8,
+    "thinker.audio_tower.layers.2.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.2.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.2.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.2.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.20.fc1": 8,
+    "thinker.audio_tower.layers.20.fc2": 8,
+    "thinker.audio_tower.layers.20.self_attn.k_proj": 24,
+    "thinker.audio_tower.layers.20.self_attn.out_proj": 24,
+    "thinker.audio_tower.layers.20.self_attn.q_proj": 24,
+    "thinker.audio_tower.layers.20.self_attn.v_proj": 24,
+    "thinker.audio_tower.layers.21.fc1": 8,
+    "thinker.audio_tower.layers.21.fc2": 8,
+    "thinker.audio_tower.layers.21.self_attn.k_proj": 24,
+    "thinker.audio_tower.layers.21.self_attn.out_proj": 24,
+    "thinker.audio_tower.layers.21.self_attn.q_proj": 24,
+    "thinker.audio_tower.layers.21.self_attn.v_proj": 24,
+    "thinker.audio_tower.layers.22.fc1": 8,
+    "thinker.audio_tower.layers.22.fc2": 8,
+    "thinker.audio_tower.layers.22.self_attn.k_proj": 24,
+    "thinker.audio_tower.layers.22.self_attn.out_proj": 24,
+    "thinker.audio_tower.layers.22.self_attn.q_proj": 24,
+    "thinker.audio_tower.layers.22.self_attn.v_proj": 24,
+    "thinker.audio_tower.layers.23.fc1": 8,
+    "thinker.audio_tower.layers.23.fc2": 8,
+    "thinker.audio_tower.layers.23.self_attn.k_proj": 24,
+    "thinker.audio_tower.layers.23.self_attn.out_proj": 24,
+    "thinker.audio_tower.layers.23.self_attn.q_proj": 24,
+    "thinker.audio_tower.layers.23.self_attn.v_proj": 24,
+    "thinker.audio_tower.layers.3.fc1": 8,
+    "thinker.audio_tower.layers.3.fc2": 8,
+    "thinker.audio_tower.layers.3.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.3.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.3.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.3.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.4.fc1": 8,
+    "thinker.audio_tower.layers.4.fc2": 8,
+    "thinker.audio_tower.layers.4.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.4.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.4.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.4.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.5.fc1": 8,
+    "thinker.audio_tower.layers.5.fc2": 8,
+    "thinker.audio_tower.layers.5.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.5.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.5.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.5.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.6.fc1": 8,
+    "thinker.audio_tower.layers.6.fc2": 8,
+    "thinker.audio_tower.layers.6.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.6.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.6.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.6.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.7.fc1": 8,
+    "thinker.audio_tower.layers.7.fc2": 8,
+    "thinker.audio_tower.layers.7.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.7.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.7.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.7.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.8.fc1": 8,
+    "thinker.audio_tower.layers.8.fc2": 8,
+    "thinker.audio_tower.layers.8.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.8.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.8.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.8.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.9.fc1": 8,
+    "thinker.audio_tower.layers.9.fc2": 8,
+    "thinker.audio_tower.layers.9.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.9.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.9.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.9.self_attn.v_proj": 16,
+    "thinker.audio_tower.proj1": 16,
+    "thinker.audio_tower.proj2": 16,
+    "thinker.layers.0.mlp.down_proj": 8,
+    "thinker.layers.0.mlp.gate_proj": 8,
+    "thinker.layers.0.mlp.up_proj": 8,
+    "thinker.layers.0.self_attn.k_proj": 8,
+    "thinker.layers.0.self_attn.o_proj": 8,
+    "thinker.layers.0.self_attn.q_proj": 8,
+    "thinker.layers.0.self_attn.v_proj": 8,
+    "thinker.layers.1.mlp.down_proj": 8,
+    "thinker.layers.1.mlp.gate_proj": 8,
+    "thinker.layers.1.mlp.up_proj": 8,
+    "thinker.layers.1.self_attn.k_proj": 8,
+    "thinker.layers.1.self_attn.o_proj": 8,
+    "thinker.layers.1.self_attn.q_proj": 8,
+    "thinker.layers.1.self_attn.v_proj": 8,
+    "thinker.layers.10.mlp.down_proj": 8,
+    "thinker.layers.10.mlp.gate_proj": 8,
+    "thinker.layers.10.mlp.up_proj": 8,
+    "thinker.layers.10.self_attn.k_proj": 8,
+    "thinker.layers.10.self_attn.o_proj": 8,
+    "thinker.layers.10.self_attn.q_proj": 8,
+    "thinker.layers.10.self_attn.v_proj": 8,
+    "thinker.layers.11.mlp.down_proj": 8,
+    "thinker.layers.11.mlp.gate_proj": 8,
+    "thinker.layers.11.mlp.up_proj": 8,
+    "thinker.layers.11.self_attn.k_proj": 8,
+    "thinker.layers.11.self_attn.o_proj": 8,
+    "thinker.layers.11.self_attn.q_proj": 8,
+    "thinker.layers.11.self_attn.v_proj": 8,
+    "thinker.layers.12.mlp.down_proj": 8,
+    "thinker.layers.12.mlp.gate_proj": 8,
+    "thinker.layers.12.mlp.up_proj": 8,
+    "thinker.layers.12.self_attn.k_proj": 8,
+    "thinker.layers.12.self_attn.o_proj": 8,
+    "thinker.layers.12.self_attn.q_proj": 8,
+    "thinker.layers.12.self_attn.v_proj": 8,
+    "thinker.layers.13.mlp.down_proj": 8,
+    "thinker.layers.13.mlp.gate_proj": 8,
+    "thinker.layers.13.mlp.up_proj": 8,
+    "thinker.layers.13.self_attn.k_proj": 8,
+    "thinker.layers.13.self_attn.o_proj": 8,
+    "thinker.layers.13.self_attn.q_proj": 8,
+    "thinker.layers.13.self_attn.v_proj": 8,
+    "thinker.layers.14.mlp.down_proj": 8,
+    "thinker.layers.14.mlp.gate_proj": 8,
+    "thinker.layers.14.mlp.up_proj": 8,
+    "thinker.layers.14.self_attn.k_proj": 8,
+    "thinker.layers.14.self_attn.o_proj": 8,
+    "thinker.layers.14.self_attn.q_proj": 8,
+    "thinker.layers.14.self_attn.v_proj": 8,
+    "thinker.layers.15.mlp.down_proj": 8,
+    "thinker.layers.15.mlp.gate_proj": 8,
+    "thinker.layers.15.mlp.up_proj": 8,
+    "thinker.layers.15.self_attn.k_proj": 8,
+    "thinker.layers.15.self_attn.o_proj": 8,
+    "thinker.layers.15.self_attn.q_proj": 8,
+    "thinker.layers.15.self_attn.v_proj": 8,
+    "thinker.layers.16.mlp.down_proj": 8,
+    "thinker.layers.16.mlp.gate_proj": 8,
+    "thinker.layers.16.mlp.up_proj": 8,
+    "thinker.layers.16.self_attn.k_proj": 8,
+    "thinker.layers.16.self_attn.o_proj": 8,
+    "thinker.layers.16.self_attn.q_proj": 8,
+    "thinker.layers.16.self_attn.v_proj": 8,
+    "thinker.layers.17.mlp.down_proj": 8,
+    "thinker.layers.17.mlp.gate_proj": 8,
+    "thinker.layers.17.mlp.up_proj": 8,
+    "thinker.layers.17.self_attn.k_proj": 8,
+    "thinker.layers.17.self_attn.o_proj": 8,
+    "thinker.layers.17.self_attn.q_proj": 8,
+    "thinker.layers.17.self_attn.v_proj": 8,
+    "thinker.layers.18.mlp.down_proj": 8,
+    "thinker.layers.18.mlp.gate_proj": 8,
+    "thinker.layers.18.mlp.up_proj": 8,
+    "thinker.layers.18.self_attn.k_proj": 8,
+    "thinker.layers.18.self_attn.o_proj": 8,
+    "thinker.layers.18.self_attn.q_proj": 8,
+    "thinker.layers.18.self_attn.v_proj": 8,
+    "thinker.layers.19.mlp.down_proj": 8,
+    "thinker.layers.19.mlp.gate_proj": 8,
+    "thinker.layers.19.mlp.up_proj": 8,
+    "thinker.layers.19.self_attn.k_proj": 8,
+    "thinker.layers.19.self_attn.o_proj": 8,
+    "thinker.layers.19.self_attn.q_proj": 8,
+    "thinker.layers.19.self_attn.v_proj": 8,
+    "thinker.layers.2.mlp.down_proj": 8,
+    "thinker.layers.2.mlp.gate_proj": 8,
+    "thinker.layers.2.mlp.up_proj": 8,
+    "thinker.layers.2.self_attn.k_proj": 8,
+    "thinker.layers.2.self_attn.o_proj": 8,
+    "thinker.layers.2.self_attn.q_proj": 8,
+    "thinker.layers.2.self_attn.v_proj": 8,
+    "thinker.layers.20.mlp.down_proj": 8,
+    "thinker.layers.20.mlp.gate_proj": 8,
+    "thinker.layers.20.mlp.up_proj": 8,
+    "thinker.layers.20.self_attn.k_proj": 8,
+    "thinker.layers.20.self_attn.o_proj": 8,
+    "thinker.layers.20.self_attn.q_proj": 8,
+    "thinker.layers.20.self_attn.v_proj": 8,
+    "thinker.layers.21.mlp.down_proj": 8,
+    "thinker.layers.21.mlp.gate_proj": 8,
+    "thinker.layers.21.mlp.up_proj": 8,
+    "thinker.layers.21.self_attn.k_proj": 8,
+    "thinker.layers.21.self_attn.o_proj": 8,
+    "thinker.layers.21.self_attn.q_proj": 8,
+    "thinker.layers.21.self_attn.v_proj": 8,
+    "thinker.layers.22.mlp.down_proj": 8,
+    "thinker.layers.22.mlp.gate_proj": 8,
+    "thinker.layers.22.mlp.up_proj": 8,
+    "thinker.layers.22.self_attn.k_proj": 8,
+    "thinker.layers.22.self_attn.o_proj": 8,
+    "thinker.layers.22.self_attn.q_proj": 8,
+    "thinker.layers.22.self_attn.v_proj": 8,
+    "thinker.layers.23.mlp.down_proj": 8,
+    "thinker.layers.23.mlp.gate_proj": 8,
+    "thinker.layers.23.mlp.up_proj": 8,
+    "thinker.layers.23.self_attn.k_proj": 8,
+    "thinker.layers.23.self_attn.o_proj": 8,
+    "thinker.layers.23.self_attn.q_proj": 8,
+    "thinker.layers.23.self_attn.v_proj": 8,
+    "thinker.layers.24.mlp.down_proj": 8,
+    "thinker.layers.24.mlp.gate_proj": 8,
+    "thinker.layers.24.mlp.up_proj": 8,
+    "thinker.layers.24.self_attn.k_proj": 8,
+    "thinker.layers.24.self_attn.o_proj": 8,
+    "thinker.layers.24.self_attn.q_proj": 8,
+    "thinker.layers.24.self_attn.v_proj": 8,
+    "thinker.layers.25.mlp.down_proj": 8,
+    "thinker.layers.25.mlp.gate_proj": 8,
+    "thinker.layers.25.mlp.up_proj": 8,
+    "thinker.layers.25.self_attn.k_proj": 8,
+    "thinker.layers.25.self_attn.o_proj": 8,
+    "thinker.layers.25.self_attn.q_proj": 8,
+    "thinker.layers.25.self_attn.v_proj": 8,
+    "thinker.layers.26.mlp.down_proj": 8,
+    "thinker.layers.26.mlp.gate_proj": 8,
+    "thinker.layers.26.mlp.up_proj": 8,
+    "thinker.layers.26.self_attn.k_proj": 8,
+    "thinker.layers.26.self_attn.o_proj": 8,
+    "thinker.layers.26.self_attn.q_proj": 8,
+    "thinker.layers.26.self_attn.v_proj": 8,
+    "thinker.layers.27.mlp.down_proj": 8,
+    "thinker.layers.27.mlp.gate_proj": 8,
+    "thinker.layers.27.mlp.up_proj": 8,
+    "thinker.layers.27.self_attn.k_proj": 8,
+    "thinker.layers.27.self_attn.o_proj": 8,
+    "thinker.layers.27.self_attn.q_proj": 8,
+    "thinker.layers.27.self_attn.v_proj": 8,
+    "thinker.layers.3.mlp.down_proj": 8,
+    "thinker.layers.3.mlp.gate_proj": 8,
+    "thinker.layers.3.mlp.up_proj": 8,
+    "thinker.layers.3.self_attn.k_proj": 8,
+    "thinker.layers.3.self_attn.o_proj": 8,
+    "thinker.layers.3.self_attn.q_proj": 8,
+    "thinker.layers.3.self_attn.v_proj": 8,
+    "thinker.layers.4.mlp.down_proj": 8,
+    "thinker.layers.4.mlp.gate_proj": 8,
+    "thinker.layers.4.mlp.up_proj": 8,
+    "thinker.layers.4.self_attn.k_proj": 8,
+    "thinker.layers.4.self_attn.o_proj": 8,
+    "thinker.layers.4.self_attn.q_proj": 8,
+    "thinker.layers.4.self_attn.v_proj": 8,
+    "thinker.layers.5.mlp.down_proj": 8,
+    "thinker.layers.5.mlp.gate_proj": 8,
+    "thinker.layers.5.mlp.up_proj": 8,
+    "thinker.layers.5.self_attn.k_proj": 8,
+    "thinker.layers.5.self_attn.o_proj": 8,
+    "thinker.layers.5.self_attn.q_proj": 8,
+    "thinker.layers.5.self_attn.v_proj": 8,
+    "thinker.layers.6.mlp.down_proj": 8,
+    "thinker.layers.6.mlp.gate_proj": 8,
+    "thinker.layers.6.mlp.up_proj": 8,
+    "thinker.layers.6.self_attn.k_proj": 8,
+    "thinker.layers.6.self_attn.o_proj": 8,
+    "thinker.layers.6.self_attn.q_proj": 8,
+    "thinker.layers.6.self_attn.v_proj": 8,
+    "thinker.layers.7.mlp.down_proj": 8,
+    "thinker.layers.7.mlp.gate_proj": 8,
+    "thinker.layers.7.mlp.up_proj": 8,
+    "thinker.layers.7.self_attn.k_proj": 8,
+    "thinker.layers.7.self_attn.o_proj": 8,
+    "thinker.layers.7.self_attn.q_proj": 8,
+    "thinker.layers.7.self_attn.v_proj": 8,
+    "thinker.layers.8.mlp.down_proj": 8,
+    "thinker.layers.8.mlp.gate_proj": 8,
+    "thinker.layers.8.mlp.up_proj": 8,
+    "thinker.layers.8.self_attn.k_proj": 8,
+    "thinker.layers.8.self_attn.o_proj": 8,
+    "thinker.layers.8.self_attn.q_proj": 8,
+    "thinker.layers.8.self_attn.v_proj": 8,
+    "thinker.layers.9.mlp.down_proj": 8,
+    "thinker.layers.9.mlp.gate_proj": 8,
+    "thinker.layers.9.mlp.up_proj": 8,
+    "thinker.layers.9.self_attn.k_proj": 8,
+    "thinker.layers.9.self_attn.o_proj": 8,
+    "thinker.layers.9.self_attn.q_proj": 8,
+    "thinker.layers.9.self_attn.v_proj": 8,
+    "thinker.model.layers.0.mlp.down_proj": 8,
+    "thinker.model.layers.0.mlp.gate_proj": 8,
+    "thinker.model.layers.0.mlp.up_proj": 8,
+    "thinker.model.layers.0.self_attn.k_proj": 8,
+    "thinker.model.layers.0.self_attn.o_proj": 8,
+    "thinker.model.layers.0.self_attn.q_proj": 8,
+    "thinker.model.layers.0.self_attn.v_proj": 8,
+    "thinker.model.layers.1.mlp.down_proj": 8,
+    "thinker.model.layers.1.mlp.gate_proj": 8,
+    "thinker.model.layers.1.mlp.up_proj": 8,
+    "thinker.model.layers.1.self_attn.k_proj": 8,
+    "thinker.model.layers.1.self_attn.o_proj": 8,
+    "thinker.model.layers.1.self_attn.q_proj": 8,
+    "thinker.model.layers.1.self_attn.v_proj": 8,
+    "thinker.model.layers.10.mlp.down_proj": 8,
+    "thinker.model.layers.10.mlp.gate_proj": 8,
+    "thinker.model.layers.10.mlp.up_proj": 8,
+    "thinker.model.layers.10.self_attn.k_proj": 8,
+    "thinker.model.layers.10.self_attn.o_proj": 8,
+    "thinker.model.layers.10.self_attn.q_proj": 8,
+    "thinker.model.layers.10.self_attn.v_proj": 8,
+    "thinker.model.layers.11.mlp.down_proj": 8,
+    "thinker.model.layers.11.mlp.gate_proj": 8,
+    "thinker.model.layers.11.mlp.up_proj": 8,
+    "thinker.model.layers.11.self_attn.k_proj": 8,
+    "thinker.model.layers.11.self_attn.o_proj": 8,
+    "thinker.model.layers.11.self_attn.q_proj": 8,
+    "thinker.model.layers.11.self_attn.v_proj": 8,
+    "thinker.model.layers.12.mlp.down_proj": 8,
+    "thinker.model.layers.12.mlp.gate_proj": 8,
+    "thinker.model.layers.12.mlp.up_proj": 8,
+    "thinker.model.layers.12.self_attn.k_proj": 8,
+    "thinker.model.layers.12.self_attn.o_proj": 8,
+    "thinker.model.layers.12.self_attn.q_proj": 8,
+    "thinker.model.layers.12.self_attn.v_proj": 8,
+    "thinker.model.layers.13.mlp.down_proj": 8,
+    "thinker.model.layers.13.mlp.gate_proj": 8,
+    "thinker.model.layers.13.mlp.up_proj": 8,
+    "thinker.model.layers.13.self_attn.k_proj": 8,
+    "thinker.model.layers.13.self_attn.o_proj": 8,
+    "thinker.model.layers.13.self_attn.q_proj": 8,
+    "thinker.model.layers.13.self_attn.v_proj": 8,
+    "thinker.model.layers.14.mlp.down_proj": 8,
+    "thinker.model.layers.14.mlp.gate_proj": 8,
+    "thinker.model.layers.14.mlp.up_proj": 8,
+    "thinker.model.layers.14.self_attn.k_proj": 8,
+    "thinker.model.layers.14.self_attn.o_proj": 8,
+    "thinker.model.layers.14.self_attn.q_proj": 8,
+    "thinker.model.layers.14.self_attn.v_proj": 8,
+    "thinker.model.layers.15.mlp.down_proj": 8,
+    "thinker.model.layers.15.mlp.gate_proj": 8,
+    "thinker.model.layers.15.mlp.up_proj": 8,
+    "thinker.model.layers.15.self_attn.k_proj": 8,
+    "thinker.model.layers.15.self_attn.o_proj": 8,
+    "thinker.model.layers.15.self_attn.q_proj": 8,
+    "thinker.model.layers.15.self_attn.v_proj": 8,
+    "thinker.model.layers.16.mlp.down_proj": 8,
+    "thinker.model.layers.16.mlp.gate_proj": 8,
+    "thinker.model.layers.16.mlp.up_proj": 8,
+    "thinker.model.layers.16.self_attn.k_proj": 8,
+    "thinker.model.layers.16.self_attn.o_proj": 8,
+    "thinker.model.layers.16.self_attn.q_proj": 8,
+    "thinker.model.layers.16.self_attn.v_proj": 8,
+    "thinker.model.layers.17.mlp.down_proj": 8,
+    "thinker.model.layers.17.mlp.gate_proj": 8,
+    "thinker.model.layers.17.mlp.up_proj": 8,
+    "thinker.model.layers.17.self_attn.k_proj": 8,
+    "thinker.model.layers.17.self_attn.o_proj": 8,
+    "thinker.model.layers.17.self_attn.q_proj": 8,
+    "thinker.model.layers.17.self_attn.v_proj": 8,
+    "thinker.model.layers.18.mlp.down_proj": 8,
+    "thinker.model.layers.18.mlp.gate_proj": 8,
+    "thinker.model.layers.18.mlp.up_proj": 8,
+    "thinker.model.layers.18.self_attn.k_proj": 8,
+    "thinker.model.layers.18.self_attn.o_proj": 8,
+    "thinker.model.layers.18.self_attn.q_proj": 8,
+    "thinker.model.layers.18.self_attn.v_proj": 8,
+    "thinker.model.layers.19.mlp.down_proj": 8,
+    "thinker.model.layers.19.mlp.gate_proj": 8,
+    "thinker.model.layers.19.mlp.up_proj": 8,
+    "thinker.model.layers.19.self_attn.k_proj": 8,
+    "thinker.model.layers.19.self_attn.o_proj": 8,
+    "thinker.model.layers.19.self_attn.q_proj": 8,
+    "thinker.model.layers.19.self_attn.v_proj": 8,
+    "thinker.model.layers.2.mlp.down_proj": 8,
+    "thinker.model.layers.2.mlp.gate_proj": 8,
+    "thinker.model.layers.2.mlp.up_proj": 8,
+    "thinker.model.layers.2.self_attn.k_proj": 8,
+    "thinker.model.layers.2.self_attn.o_proj": 8,
+    "thinker.model.layers.2.self_attn.q_proj": 8,
+    "thinker.model.layers.2.self_attn.v_proj": 8,
+    "thinker.model.layers.20.mlp.down_proj": 8,
+    "thinker.model.layers.20.mlp.gate_proj": 8,
+    "thinker.model.layers.20.mlp.up_proj": 8,
+    "thinker.model.layers.20.self_attn.k_proj": 8,
+    "thinker.model.layers.20.self_attn.o_proj": 8,
+    "thinker.model.layers.20.self_attn.q_proj": 8,
+    "thinker.model.layers.20.self_attn.v_proj": 8,
+    "thinker.model.layers.21.mlp.down_proj": 8,
+    "thinker.model.layers.21.mlp.gate_proj": 8,
+    "thinker.model.layers.21.mlp.up_proj": 8,
+    "thinker.model.layers.21.self_attn.k_proj": 8,
+    "thinker.model.layers.21.self_attn.o_proj": 8,
+    "thinker.model.layers.21.self_attn.q_proj": 8,
+    "thinker.model.layers.21.self_attn.v_proj": 8,
+    "thinker.model.layers.22.mlp.down_proj": 8,
+    "thinker.model.layers.22.mlp.gate_proj": 8,
+    "thinker.model.layers.22.mlp.up_proj": 8,
+    "thinker.model.layers.22.self_attn.k_proj": 8,
+    "thinker.model.layers.22.self_attn.o_proj": 8,
+    "thinker.model.layers.22.self_attn.q_proj": 8,
+    "thinker.model.layers.22.self_attn.v_proj": 8,
+    "thinker.model.layers.23.mlp.down_proj": 8,
+    "thinker.model.layers.23.mlp.gate_proj": 8,
+    "thinker.model.layers.23.mlp.up_proj": 8,
+    "thinker.model.layers.23.self_attn.k_proj": 8,
+    "thinker.model.layers.23.self_attn.o_proj": 8,
+    "thinker.model.layers.23.self_attn.q_proj": 8,
+    "thinker.model.layers.23.self_attn.v_proj": 8,
+    "thinker.model.layers.24.mlp.down_proj": 8,
+    "thinker.model.layers.24.mlp.gate_proj": 8,
+    "thinker.model.layers.24.mlp.up_proj": 8,
+    "thinker.model.layers.24.self_attn.k_proj": 8,
+    "thinker.model.layers.24.self_attn.o_proj": 8,
+    "thinker.model.layers.24.self_attn.q_proj": 8,
+    "thinker.model.layers.24.self_attn.v_proj": 8,
+    "thinker.model.layers.25.mlp.down_proj": 8,
+    "thinker.model.layers.25.mlp.gate_proj": 8,
+    "thinker.model.layers.25.mlp.up_proj": 8,
+    "thinker.model.layers.25.self_attn.k_proj": 8,
+    "thinker.model.layers.25.self_attn.o_proj": 8,
+    "thinker.model.layers.25.self_attn.q_proj": 8,
+    "thinker.model.layers.25.self_attn.v_proj": 8,
+    "thinker.model.layers.26.mlp.down_proj": 8,
+    "thinker.model.layers.26.mlp.gate_proj": 8,
+    "thinker.model.layers.26.mlp.up_proj": 8,
+    "thinker.model.layers.26.self_attn.k_proj": 8,
+    "thinker.model.layers.26.self_attn.o_proj": 8,
+    "thinker.model.layers.26.self_attn.q_proj": 8,
+    "thinker.model.layers.26.self_attn.v_proj": 8,
+    "thinker.model.layers.27.mlp.down_proj": 8,
+    "thinker.model.layers.27.mlp.gate_proj": 8,
+    "thinker.model.layers.27.mlp.up_proj": 8,
+    "thinker.model.layers.27.self_attn.k_proj": 8,
+    "thinker.model.layers.27.self_attn.o_proj": 8,
+    "thinker.model.layers.27.self_attn.q_proj": 8,
+    "thinker.model.layers.27.self_attn.v_proj": 8,
+    "thinker.model.layers.3.mlp.down_proj": 8,
+    "thinker.model.layers.3.mlp.gate_proj": 8,
+    "thinker.model.layers.3.mlp.up_proj": 8,
+    "thinker.model.layers.3.self_attn.k_proj": 8,
+    "thinker.model.layers.3.self_attn.o_proj": 8,
+    "thinker.model.layers.3.self_attn.q_proj": 8,
+    "thinker.model.layers.3.self_attn.v_proj": 8,
+    "thinker.model.layers.4.mlp.down_proj": 8,
+    "thinker.model.layers.4.mlp.gate_proj": 8,
+    "thinker.model.layers.4.mlp.up_proj": 8,
+    "thinker.model.layers.4.self_attn.k_proj": 8,
+    "thinker.model.layers.4.self_attn.o_proj": 8,
+    "thinker.model.layers.4.self_attn.q_proj": 8,
+    "thinker.model.layers.4.self_attn.v_proj": 8,
+    "thinker.model.layers.5.mlp.down_proj": 8,
+    "thinker.model.layers.5.mlp.gate_proj": 8,
+    "thinker.model.layers.5.mlp.up_proj": 8,
+    "thinker.model.layers.5.self_attn.k_proj": 8,
+    "thinker.model.layers.5.self_attn.o_proj": 8,
+    "thinker.model.layers.5.self_attn.q_proj": 8,
+    "thinker.model.layers.5.self_attn.v_proj": 8,
+    "thinker.model.layers.6.mlp.down_proj": 8,
+    "thinker.model.layers.6.mlp.gate_proj": 8,
+    "thinker.model.layers.6.mlp.up_proj": 8,
+    "thinker.model.layers.6.self_attn.k_proj": 8,
+    "thinker.model.layers.6.self_attn.o_proj": 8,
+    "thinker.model.layers.6.self_attn.q_proj": 8,
+    "thinker.model.layers.6.self_attn.v_proj": 8,
+    "thinker.model.layers.7.mlp.down_proj": 8,
+    "thinker.model.layers.7.mlp.gate_proj": 8,
+    "thinker.model.layers.7.mlp.up_proj": 8,
+    "thinker.model.layers.7.self_attn.k_proj": 8,
+    "thinker.model.layers.7.self_attn.o_proj": 8,
+    "thinker.model.layers.7.self_attn.q_proj": 8,
+    "thinker.model.layers.7.self_attn.v_proj": 8,
+    "thinker.model.layers.8.mlp.down_proj": 8,
+    "thinker.model.layers.8.mlp.gate_proj": 8,
+    "thinker.model.layers.8.mlp.up_proj": 8,
+    "thinker.model.layers.8.self_attn.k_proj": 8,
+    "thinker.model.layers.8.self_attn.o_proj": 8,
+    "thinker.model.layers.8.self_attn.q_proj": 8,
+    "thinker.model.layers.8.self_attn.v_proj": 8,
+    "thinker.model.layers.9.mlp.down_proj": 8,
+    "thinker.model.layers.9.mlp.gate_proj": 8,
+    "thinker.model.layers.9.mlp.up_proj": 8,
+    "thinker.model.layers.9.self_attn.k_proj": 8,
+    "thinker.model.layers.9.self_attn.o_proj": 8,
+    "thinker.model.layers.9.self_attn.q_proj": 8,
+    "thinker.model.layers.9.self_attn.v_proj": 8
+  },
+  "base_model_name_or_path": "",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "lora_alpha": 24,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 24,
+  "rank_pattern": {
+    "thinker.audio_tower.conv_out": 24,
+    "thinker.audio_tower.layers.0.fc1": 8,
+    "thinker.audio_tower.layers.0.fc2": 8,
+    "thinker.audio_tower.layers.0.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.0.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.0.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.0.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.1.fc1": 8,
+    "thinker.audio_tower.layers.1.fc2": 8,
+    "thinker.audio_tower.layers.1.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.1.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.1.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.1.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.10.fc1": 8,
+    "thinker.audio_tower.layers.10.fc2": 8,
+    "thinker.audio_tower.layers.10.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.10.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.10.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.10.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.11.fc1": 8,
+    "thinker.audio_tower.layers.11.fc2": 8,
+    "thinker.audio_tower.layers.11.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.11.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.11.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.11.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.12.fc1": 8,
+    "thinker.audio_tower.layers.12.fc2": 8,
+    "thinker.audio_tower.layers.12.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.12.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.12.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.12.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.13.fc1": 8,
+    "thinker.audio_tower.layers.13.fc2": 8,
+    "thinker.audio_tower.layers.13.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.13.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.13.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.13.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.14.fc1": 8,
+    "thinker.audio_tower.layers.14.fc2": 8,
+    "thinker.audio_tower.layers.14.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.14.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.14.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.14.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.15.fc1": 8,
+    "thinker.audio_tower.layers.15.fc2": 8,
+    "thinker.audio_tower.layers.15.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.15.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.15.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.15.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.16.fc1": 8,
+    "thinker.audio_tower.layers.16.fc2": 8,
+    "thinker.audio_tower.layers.16.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.16.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.16.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.16.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.17.fc1": 8,
+    "thinker.audio_tower.layers.17.fc2": 8,
+    "thinker.audio_tower.layers.17.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.17.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.17.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.17.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.18.fc1": 8,
+    "thinker.audio_tower.layers.18.fc2": 8,
+    "thinker.audio_tower.layers.18.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.18.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.18.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.18.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.19.fc1": 8,
+    "thinker.audio_tower.layers.19.fc2": 8,
+    "thinker.audio_tower.layers.19.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.19.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.19.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.19.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.2.fc1": 8,
+    "thinker.audio_tower.layers.2.fc2": 8,
+    "thinker.audio_tower.layers.2.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.2.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.2.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.2.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.20.fc1": 8,
+    "thinker.audio_tower.layers.20.fc2": 8,
+    "thinker.audio_tower.layers.20.self_attn.k_proj": 24,
+    "thinker.audio_tower.layers.20.self_attn.out_proj": 24,
+    "thinker.audio_tower.layers.20.self_attn.q_proj": 24,
+    "thinker.audio_tower.layers.20.self_attn.v_proj": 24,
+    "thinker.audio_tower.layers.21.fc1": 8,
+    "thinker.audio_tower.layers.21.fc2": 8,
+    "thinker.audio_tower.layers.21.self_attn.k_proj": 24,
+    "thinker.audio_tower.layers.21.self_attn.out_proj": 24,
+    "thinker.audio_tower.layers.21.self_attn.q_proj": 24,
+    "thinker.audio_tower.layers.21.self_attn.v_proj": 24,
+    "thinker.audio_tower.layers.22.fc1": 8,
+    "thinker.audio_tower.layers.22.fc2": 8,
+    "thinker.audio_tower.layers.22.self_attn.k_proj": 24,
+    "thinker.audio_tower.layers.22.self_attn.out_proj": 24,
+    "thinker.audio_tower.layers.22.self_attn.q_proj": 24,
+    "thinker.audio_tower.layers.22.self_attn.v_proj": 24,
+    "thinker.audio_tower.layers.23.fc1": 8,
+    "thinker.audio_tower.layers.23.fc2": 8,
+    "thinker.audio_tower.layers.23.self_attn.k_proj": 24,
+    "thinker.audio_tower.layers.23.self_attn.out_proj": 24,
+    "thinker.audio_tower.layers.23.self_attn.q_proj": 24,
+    "thinker.audio_tower.layers.23.self_attn.v_proj": 24,
+    "thinker.audio_tower.layers.3.fc1": 8,
+    "thinker.audio_tower.layers.3.fc2": 8,
+    "thinker.audio_tower.layers.3.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.3.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.3.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.3.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.4.fc1": 8,
+    "thinker.audio_tower.layers.4.fc2": 8,
+    "thinker.audio_tower.layers.4.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.4.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.4.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.4.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.5.fc1": 8,
+    "thinker.audio_tower.layers.5.fc2": 8,
+    "thinker.audio_tower.layers.5.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.5.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.5.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.5.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.6.fc1": 8,
+    "thinker.audio_tower.layers.6.fc2": 8,
+    "thinker.audio_tower.layers.6.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.6.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.6.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.6.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.7.fc1": 8,
+    "thinker.audio_tower.layers.7.fc2": 8,
+    "thinker.audio_tower.layers.7.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.7.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.7.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.7.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.8.fc1": 8,
+    "thinker.audio_tower.layers.8.fc2": 8,
+    "thinker.audio_tower.layers.8.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.8.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.8.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.8.self_attn.v_proj": 16,
+    "thinker.audio_tower.layers.9.fc1": 8,
+    "thinker.audio_tower.layers.9.fc2": 8,
+    "thinker.audio_tower.layers.9.self_attn.k_proj": 16,
+    "thinker.audio_tower.layers.9.self_attn.out_proj": 16,
+    "thinker.audio_tower.layers.9.self_attn.q_proj": 16,
+    "thinker.audio_tower.layers.9.self_attn.v_proj": 16,
+    "thinker.audio_tower.proj1": 16,
+    "thinker.audio_tower.proj2": 16,
+    "thinker.layers.0.mlp.down_proj": 8,
+    "thinker.layers.0.mlp.gate_proj": 8,
+    "thinker.layers.0.mlp.up_proj": 8,
+    "thinker.layers.0.self_attn.k_proj": 8,
+    "thinker.layers.0.self_attn.o_proj": 8,
+    "thinker.layers.0.self_attn.q_proj": 8,
+    "thinker.layers.0.self_attn.v_proj": 8,
+    "thinker.layers.1.mlp.down_proj": 8,
+    "thinker.layers.1.mlp.gate_proj": 8,
+    "thinker.layers.1.mlp.up_proj": 8,
+    "thinker.layers.1.self_attn.k_proj": 8,
+    "thinker.layers.1.self_attn.o_proj": 8,
+    "thinker.layers.1.self_attn.q_proj": 8,
+    "thinker.layers.1.self_attn.v_proj": 8,
+    "thinker.layers.10.mlp.down_proj": 8,
+    "thinker.layers.10.mlp.gate_proj": 8,
+    "thinker.layers.10.mlp.up_proj": 8,
+    "thinker.layers.10.self_attn.k_proj": 8,
+    "thinker.layers.10.self_attn.o_proj": 8,
+    "thinker.layers.10.self_attn.q_proj": 8,
+    "thinker.layers.10.self_attn.v_proj": 8,
+    "thinker.layers.11.mlp.down_proj": 8,
+    "thinker.layers.11.mlp.gate_proj": 8,
+    "thinker.layers.11.mlp.up_proj": 8,
+    "thinker.layers.11.self_attn.k_proj": 8,
+    "thinker.layers.11.self_attn.o_proj": 8,
+    "thinker.layers.11.self_attn.q_proj": 8,
+    "thinker.layers.11.self_attn.v_proj": 8,
+    "thinker.layers.12.mlp.down_proj": 8,
+    "thinker.layers.12.mlp.gate_proj": 8,
+    "thinker.layers.12.mlp.up_proj": 8,
+    "thinker.layers.12.self_attn.k_proj": 8,
+    "thinker.layers.12.self_attn.o_proj": 8,
+    "thinker.layers.12.self_attn.q_proj": 8,
+    "thinker.layers.12.self_attn.v_proj": 8,
+    "thinker.layers.13.mlp.down_proj": 8,
+    "thinker.layers.13.mlp.gate_proj": 8,
+    "thinker.layers.13.mlp.up_proj": 8,
+    "thinker.layers.13.self_attn.k_proj": 8,
+    "thinker.layers.13.self_attn.o_proj": 8,
+    "thinker.layers.13.self_attn.q_proj": 8,
+    "thinker.layers.13.self_attn.v_proj": 8,
+    "thinker.layers.14.mlp.down_proj": 8,
+    "thinker.layers.14.mlp.gate_proj": 8,
+    "thinker.layers.14.mlp.up_proj": 8,
+    "thinker.layers.14.self_attn.k_proj": 8,
+    "thinker.layers.14.self_attn.o_proj": 8,
+    "thinker.layers.14.self_attn.q_proj": 8,
+    "thinker.layers.14.self_attn.v_proj": 8,
+    "thinker.layers.15.mlp.down_proj": 8,
+    "thinker.layers.15.mlp.gate_proj": 8,
+    "thinker.layers.15.mlp.up_proj": 8,
+    "thinker.layers.15.self_attn.k_proj": 8,
+    "thinker.layers.15.self_attn.o_proj": 8,
+    "thinker.layers.15.self_attn.q_proj": 8,
+    "thinker.layers.15.self_attn.v_proj": 8,
+    "thinker.layers.16.mlp.down_proj": 8,
+    "thinker.layers.16.mlp.gate_proj": 8,
+    "thinker.layers.16.mlp.up_proj": 8,
+    "thinker.layers.16.self_attn.k_proj": 8,
+    "thinker.layers.16.self_attn.o_proj": 8,
+    "thinker.layers.16.self_attn.q_proj": 8,
+    "thinker.layers.16.self_attn.v_proj": 8,
+    "thinker.layers.17.mlp.down_proj": 8,
+    "thinker.layers.17.mlp.gate_proj": 8,
+    "thinker.layers.17.mlp.up_proj": 8,
+    "thinker.layers.17.self_attn.k_proj": 8,
+    "thinker.layers.17.self_attn.o_proj": 8,
+    "thinker.layers.17.self_attn.q_proj": 8,
+    "thinker.layers.17.self_attn.v_proj": 8,
+    "thinker.layers.18.mlp.down_proj": 8,
+    "thinker.layers.18.mlp.gate_proj": 8,
+    "thinker.layers.18.mlp.up_proj": 8,
+    "thinker.layers.18.self_attn.k_proj": 8,
+    "thinker.layers.18.self_attn.o_proj": 8,
+    "thinker.layers.18.self_attn.q_proj": 8,
+    "thinker.layers.18.self_attn.v_proj": 8,
+    "thinker.layers.19.mlp.down_proj": 8,
+    "thinker.layers.19.mlp.gate_proj": 8,
+    "thinker.layers.19.mlp.up_proj": 8,
+    "thinker.layers.19.self_attn.k_proj": 8,
+    "thinker.layers.19.self_attn.o_proj": 8,
+    "thinker.layers.19.self_attn.q_proj": 8,
+    "thinker.layers.19.self_attn.v_proj": 8,
+    "thinker.layers.2.mlp.down_proj": 8,
+    "thinker.layers.2.mlp.gate_proj": 8,
+    "thinker.layers.2.mlp.up_proj": 8,
+    "thinker.layers.2.self_attn.k_proj": 8,
+    "thinker.layers.2.self_attn.o_proj": 8,
+    "thinker.layers.2.self_attn.q_proj": 8,
+    "thinker.layers.2.self_attn.v_proj": 8,
+    "thinker.layers.20.mlp.down_proj": 8,
+    "thinker.layers.20.mlp.gate_proj": 8,
+    "thinker.layers.20.mlp.up_proj": 8,
+    "thinker.layers.20.self_attn.k_proj": 8,
+    "thinker.layers.20.self_attn.o_proj": 8,
+    "thinker.layers.20.self_attn.q_proj": 8,
+    "thinker.layers.20.self_attn.v_proj": 8,
+    "thinker.layers.21.mlp.down_proj": 8,
+    "thinker.layers.21.mlp.gate_proj": 8,
+    "thinker.layers.21.mlp.up_proj": 8,
+    "thinker.layers.21.self_attn.k_proj": 8,
+    "thinker.layers.21.self_attn.o_proj": 8,
+    "thinker.layers.21.self_attn.q_proj": 8,
+    "thinker.layers.21.self_attn.v_proj": 8,
+    "thinker.layers.22.mlp.down_proj": 8,
+    "thinker.layers.22.mlp.gate_proj": 8,
+    "thinker.layers.22.mlp.up_proj": 8,
+    "thinker.layers.22.self_attn.k_proj": 8,
+    "thinker.layers.22.self_attn.o_proj": 8,
+    "thinker.layers.22.self_attn.q_proj": 8,
+    "thinker.layers.22.self_attn.v_proj": 8,
+    "thinker.layers.23.mlp.down_proj": 8,
+    "thinker.layers.23.mlp.gate_proj": 8,
+    "thinker.layers.23.mlp.up_proj": 8,
+    "thinker.layers.23.self_attn.k_proj": 8,
+    "thinker.layers.23.self_attn.o_proj": 8,
+    "thinker.layers.23.self_attn.q_proj": 8,
+    "thinker.layers.23.self_attn.v_proj": 8,
+    "thinker.layers.24.mlp.down_proj": 8,
+    "thinker.layers.24.mlp.gate_proj": 8,
+    "thinker.layers.24.mlp.up_proj": 8,
+    "thinker.layers.24.self_attn.k_proj": 8,
+    "thinker.layers.24.self_attn.o_proj": 8,
+    "thinker.layers.24.self_attn.q_proj": 8,
+    "thinker.layers.24.self_attn.v_proj": 8,
+    "thinker.layers.25.mlp.down_proj": 8,
+    "thinker.layers.25.mlp.gate_proj": 8,
+    "thinker.layers.25.mlp.up_proj": 8,
+    "thinker.layers.25.self_attn.k_proj": 8,
+    "thinker.layers.25.self_attn.o_proj": 8,
+    "thinker.layers.25.self_attn.q_proj": 8,
+    "thinker.layers.25.self_attn.v_proj": 8,
+    "thinker.layers.26.mlp.down_proj": 8,
+    "thinker.layers.26.mlp.gate_proj": 8,
+    "thinker.layers.26.mlp.up_proj": 8,
+    "thinker.layers.26.self_attn.k_proj": 8,
+    "thinker.layers.26.self_attn.o_proj": 8,
+    "thinker.layers.26.self_attn.q_proj": 8,
+    "thinker.layers.26.self_attn.v_proj": 8,
+    "thinker.layers.27.mlp.down_proj": 8,
+    "thinker.layers.27.mlp.gate_proj": 8,
+    "thinker.layers.27.mlp.up_proj": 8,
+    "thinker.layers.27.self_attn.k_proj": 8,
+    "thinker.layers.27.self_attn.o_proj": 8,
+    "thinker.layers.27.self_attn.q_proj": 8,
+    "thinker.layers.27.self_attn.v_proj": 8,
+    "thinker.layers.3.mlp.down_proj": 8,
+    "thinker.layers.3.mlp.gate_proj": 8,
+    "thinker.layers.3.mlp.up_proj": 8,
+    "thinker.layers.3.self_attn.k_proj": 8,
+    "thinker.layers.3.self_attn.o_proj": 8,
+    "thinker.layers.3.self_attn.q_proj": 8,
+    "thinker.layers.3.self_attn.v_proj": 8,
+    "thinker.layers.4.mlp.down_proj": 8,
+    "thinker.layers.4.mlp.gate_proj": 8,
+    "thinker.layers.4.mlp.up_proj": 8,
+    "thinker.layers.4.self_attn.k_proj": 8,
+    "thinker.layers.4.self_attn.o_proj": 8,
+    "thinker.layers.4.self_attn.q_proj": 8,
+    "thinker.layers.4.self_attn.v_proj": 8,
+    "thinker.layers.5.mlp.down_proj": 8,
+    "thinker.layers.5.mlp.gate_proj": 8,
+    "thinker.layers.5.mlp.up_proj": 8,
+    "thinker.layers.5.self_attn.k_proj": 8,
+    "thinker.layers.5.self_attn.o_proj": 8,
+    "thinker.layers.5.self_attn.q_proj": 8,
+    "thinker.layers.5.self_attn.v_proj": 8,
+    "thinker.layers.6.mlp.down_proj": 8,
+    "thinker.layers.6.mlp.gate_proj": 8,
+    "thinker.layers.6.mlp.up_proj": 8,
+    "thinker.layers.6.self_attn.k_proj": 8,
+    "thinker.layers.6.self_attn.o_proj": 8,
+    "thinker.layers.6.self_attn.q_proj": 8,
+    "thinker.layers.6.self_attn.v_proj": 8,
+    "thinker.layers.7.mlp.down_proj": 8,
+    "thinker.layers.7.mlp.gate_proj": 8,
+    "thinker.layers.7.mlp.up_proj": 8,
+    "thinker.layers.7.self_attn.k_proj": 8,
+    "thinker.layers.7.self_attn.o_proj": 8,
+    "thinker.layers.7.self_attn.q_proj": 8,
+    "thinker.layers.7.self_attn.v_proj": 8,
+    "thinker.layers.8.mlp.down_proj": 8,
+    "thinker.layers.8.mlp.gate_proj": 8,
+    "thinker.layers.8.mlp.up_proj": 8,
+    "thinker.layers.8.self_attn.k_proj": 8,
+    "thinker.layers.8.self_attn.o_proj": 8,
+    "thinker.layers.8.self_attn.q_proj": 8,
+    "thinker.layers.8.self_attn.v_proj": 8,
+    "thinker.layers.9.mlp.down_proj": 8,
+    "thinker.layers.9.mlp.gate_proj": 8,
+    "thinker.layers.9.mlp.up_proj": 8,
+    "thinker.layers.9.self_attn.k_proj": 8,
+    "thinker.layers.9.self_attn.o_proj": 8,
+    "thinker.layers.9.self_attn.q_proj": 8,
+    "thinker.layers.9.self_attn.v_proj": 8,
+    "thinker.model.layers.0.mlp.down_proj": 8,
+    "thinker.model.layers.0.mlp.gate_proj": 8,
+    "thinker.model.layers.0.mlp.up_proj": 8,
+    "thinker.model.layers.0.self_attn.k_proj": 8,
+    "thinker.model.layers.0.self_attn.o_proj": 8,
+    "thinker.model.layers.0.self_attn.q_proj": 8,
+    "thinker.model.layers.0.self_attn.v_proj": 8,
+    "thinker.model.layers.1.mlp.down_proj": 8,
+    "thinker.model.layers.1.mlp.gate_proj": 8,
+    "thinker.model.layers.1.mlp.up_proj": 8,
+    "thinker.model.layers.1.self_attn.k_proj": 8,
+    "thinker.model.layers.1.self_attn.o_proj": 8,
+    "thinker.model.layers.1.self_attn.q_proj": 8,
+    "thinker.model.layers.1.self_attn.v_proj": 8,
+    "thinker.model.layers.10.mlp.down_proj": 8,
+    "thinker.model.layers.10.mlp.gate_proj": 8,
+    "thinker.model.layers.10.mlp.up_proj": 8,
+    "thinker.model.layers.10.self_attn.k_proj": 8,
+    "thinker.model.layers.10.self_attn.o_proj": 8,
+    "thinker.model.layers.10.self_attn.q_proj": 8,
+    "thinker.model.layers.10.self_attn.v_proj": 8,
+    "thinker.model.layers.11.mlp.down_proj": 8,
+    "thinker.model.layers.11.mlp.gate_proj": 8,
+    "thinker.model.layers.11.mlp.up_proj": 8,
+    "thinker.model.layers.11.self_attn.k_proj": 8,
+    "thinker.model.layers.11.self_attn.o_proj": 8,
+    "thinker.model.layers.11.self_attn.q_proj": 8,
+    "thinker.model.layers.11.self_attn.v_proj": 8,
+    "thinker.model.layers.12.mlp.down_proj": 8,
+    "thinker.model.layers.12.mlp.gate_proj": 8,
+    "thinker.model.layers.12.mlp.up_proj": 8,
+    "thinker.model.layers.12.self_attn.k_proj": 8,
+    "thinker.model.layers.12.self_attn.o_proj": 8,
+    "thinker.model.layers.12.self_attn.q_proj": 8,
+    "thinker.model.layers.12.self_attn.v_proj": 8,
+    "thinker.model.layers.13.mlp.down_proj": 8,
+    "thinker.model.layers.13.mlp.gate_proj": 8,
+    "thinker.model.layers.13.mlp.up_proj": 8,
+    "thinker.model.layers.13.self_attn.k_proj": 8,
+    "thinker.model.layers.13.self_attn.o_proj": 8,
+    "thinker.model.layers.13.self_attn.q_proj": 8,
+    "thinker.model.layers.13.self_attn.v_proj": 8,
+    "thinker.model.layers.14.mlp.down_proj": 8,
+    "thinker.model.layers.14.mlp.gate_proj": 8,
+    "thinker.model.layers.14.mlp.up_proj": 8,
+    "thinker.model.layers.14.self_attn.k_proj": 8,
+    "thinker.model.layers.14.self_attn.o_proj": 8,
+    "thinker.model.layers.14.self_attn.q_proj": 8,
+    "thinker.model.layers.14.self_attn.v_proj": 8,
+    "thinker.model.layers.15.mlp.down_proj": 8,
+    "thinker.model.layers.15.mlp.gate_proj": 8,
+    "thinker.model.layers.15.mlp.up_proj": 8,
+    "thinker.model.layers.15.self_attn.k_proj": 8,
+    "thinker.model.layers.15.self_attn.o_proj": 8,
+    "thinker.model.layers.15.self_attn.q_proj": 8,
+    "thinker.model.layers.15.self_attn.v_proj": 8,
+    "thinker.model.layers.16.mlp.down_proj": 8,
+    "thinker.model.layers.16.mlp.gate_proj": 8,
+    "thinker.model.layers.16.mlp.up_proj": 8,
+    "thinker.model.layers.16.self_attn.k_proj": 8,
+    "thinker.model.layers.16.self_attn.o_proj": 8,
+    "thinker.model.layers.16.self_attn.q_proj": 8,
+    "thinker.model.layers.16.self_attn.v_proj": 8,
+    "thinker.model.layers.17.mlp.down_proj": 8,
+    "thinker.model.layers.17.mlp.gate_proj": 8,
+    "thinker.model.layers.17.mlp.up_proj": 8,
+    "thinker.model.layers.17.self_attn.k_proj": 8,
+    "thinker.model.layers.17.self_attn.o_proj": 8,
+    "thinker.model.layers.17.self_attn.q_proj": 8,
+    "thinker.model.layers.17.self_attn.v_proj": 8,
+    "thinker.model.layers.18.mlp.down_proj": 8,
+    "thinker.model.layers.18.mlp.gate_proj": 8,
+    "thinker.model.layers.18.mlp.up_proj": 8,
+    "thinker.model.layers.18.self_attn.k_proj": 8,
+    "thinker.model.layers.18.self_attn.o_proj": 8,
+    "thinker.model.layers.18.self_attn.q_proj": 8,
+    "thinker.model.layers.18.self_attn.v_proj": 8,
+    "thinker.model.layers.19.mlp.down_proj": 8,
+    "thinker.model.layers.19.mlp.gate_proj": 8,
+    "thinker.model.layers.19.mlp.up_proj": 8,
+    "thinker.model.layers.19.self_attn.k_proj": 8,
+    "thinker.model.layers.19.self_attn.o_proj": 8,
+    "thinker.model.layers.19.self_attn.q_proj": 8,
+    "thinker.model.layers.19.self_attn.v_proj": 8,
+    "thinker.model.layers.2.mlp.down_proj": 8,
+    "thinker.model.layers.2.mlp.gate_proj": 8,
+    "thinker.model.layers.2.mlp.up_proj": 8,
+    "thinker.model.layers.2.self_attn.k_proj": 8,
+    "thinker.model.layers.2.self_attn.o_proj": 8,
+    "thinker.model.layers.2.self_attn.q_proj": 8,
+    "thinker.model.layers.2.self_attn.v_proj": 8,
+    "thinker.model.layers.20.mlp.down_proj": 8,
+    "thinker.model.layers.20.mlp.gate_proj": 8,
+    "thinker.model.layers.20.mlp.up_proj": 8,
+    "thinker.model.layers.20.self_attn.k_proj": 8,
+    "thinker.model.layers.20.self_attn.o_proj": 8,
+    "thinker.model.layers.20.self_attn.q_proj": 8,
+    "thinker.model.layers.20.self_attn.v_proj": 8,
+    "thinker.model.layers.21.mlp.down_proj": 8,
+    "thinker.model.layers.21.mlp.gate_proj": 8,
+    "thinker.model.layers.21.mlp.up_proj": 8,
+    "thinker.model.layers.21.self_attn.k_proj": 8,
+    "thinker.model.layers.21.self_attn.o_proj": 8,
+    "thinker.model.layers.21.self_attn.q_proj": 8,
+    "thinker.model.layers.21.self_attn.v_proj": 8,
+    "thinker.model.layers.22.mlp.down_proj": 8,
+    "thinker.model.layers.22.mlp.gate_proj": 8,
+    "thinker.model.layers.22.mlp.up_proj": 8,
+    "thinker.model.layers.22.self_attn.k_proj": 8,
+    "thinker.model.layers.22.self_attn.o_proj": 8,
+    "thinker.model.layers.22.self_attn.q_proj": 8,
+    "thinker.model.layers.22.self_attn.v_proj": 8,
+    "thinker.model.layers.23.mlp.down_proj": 8,
+    "thinker.model.layers.23.mlp.gate_proj": 8,
+    "thinker.model.layers.23.mlp.up_proj": 8,
+    "thinker.model.layers.23.self_attn.k_proj": 8,
+    "thinker.model.layers.23.self_attn.o_proj": 8,
+    "thinker.model.layers.23.self_attn.q_proj": 8,
+    "thinker.model.layers.23.self_attn.v_proj": 8,
+    "thinker.model.layers.24.mlp.down_proj": 8,
+    "thinker.model.layers.24.mlp.gate_proj": 8,
+    "thinker.model.layers.24.mlp.up_proj": 8,
+    "thinker.model.layers.24.self_attn.k_proj": 8,
+    "thinker.model.layers.24.self_attn.o_proj": 8,
+    "thinker.model.layers.24.self_attn.q_proj": 8,
+    "thinker.model.layers.24.self_attn.v_proj": 8,
+    "thinker.model.layers.25.mlp.down_proj": 8,
+    "thinker.model.layers.25.mlp.gate_proj": 8,
+    "thinker.model.layers.25.mlp.up_proj": 8,
+    "thinker.model.layers.25.self_attn.k_proj": 8,
+    "thinker.model.layers.25.self_attn.o_proj": 8,
+    "thinker.model.layers.25.self_attn.q_proj": 8,
+    "thinker.model.layers.25.self_attn.v_proj": 8,
+    "thinker.model.layers.26.mlp.down_proj": 8,
+    "thinker.model.layers.26.mlp.gate_proj": 8,
+    "thinker.model.layers.26.mlp.up_proj": 8,
+    "thinker.model.layers.26.self_attn.k_proj": 8,
+    "thinker.model.layers.26.self_attn.o_proj": 8,
+    "thinker.model.layers.26.self_attn.q_proj": 8,
+    "thinker.model.layers.26.self_attn.v_proj": 8,
+    "thinker.model.layers.27.mlp.down_proj": 8,
+    "thinker.model.layers.27.mlp.gate_proj": 8,
+    "thinker.model.layers.27.mlp.up_proj": 8,
+    "thinker.model.layers.27.self_attn.k_proj": 8,
+    "thinker.model.layers.27.self_attn.o_proj": 8,
+    "thinker.model.layers.27.self_attn.q_proj": 8,
+    "thinker.model.layers.27.self_attn.v_proj": 8,
+    "thinker.model.layers.3.mlp.down_proj": 8,
+    "thinker.model.layers.3.mlp.gate_proj": 8,
+    "thinker.model.layers.3.mlp.up_proj": 8,
+    "thinker.model.layers.3.self_attn.k_proj": 8,
+    "thinker.model.layers.3.self_attn.o_proj": 8,
+    "thinker.model.layers.3.self_attn.q_proj": 8,
+    "thinker.model.layers.3.self_attn.v_proj": 8,
+    "thinker.model.layers.4.mlp.down_proj": 8,
+    "thinker.model.layers.4.mlp.gate_proj": 8,
+    "thinker.model.layers.4.mlp.up_proj": 8,
+    "thinker.model.layers.4.self_attn.k_proj": 8,
+    "thinker.model.layers.4.self_attn.o_proj": 8,
+    "thinker.model.layers.4.self_attn.q_proj": 8,
+    "thinker.model.layers.4.self_attn.v_proj": 8,
+    "thinker.model.layers.5.mlp.down_proj": 8,
+    "thinker.model.layers.5.mlp.gate_proj": 8,
+    "thinker.model.layers.5.mlp.up_proj": 8,
+    "thinker.model.layers.5.self_attn.k_proj": 8,
+    "thinker.model.layers.5.self_attn.o_proj": 8,
+    "thinker.model.layers.5.self_attn.q_proj": 8,
+    "thinker.model.layers.5.self_attn.v_proj": 8,
+    "thinker.model.layers.6.mlp.down_proj": 8,
+    "thinker.model.layers.6.mlp.gate_proj": 8,
+    "thinker.model.layers.6.mlp.up_proj": 8,
+    "thinker.model.layers.6.self_attn.k_proj": 8,
+    "thinker.model.layers.6.self_attn.o_proj": 8,
+    "thinker.model.layers.6.self_attn.q_proj": 8,
+    "thinker.model.layers.6.self_attn.v_proj": 8,
+    "thinker.model.layers.7.mlp.down_proj": 8,
+    "thinker.model.layers.7.mlp.gate_proj": 8,
+    "thinker.model.layers.7.mlp.up_proj": 8,
+    "thinker.model.layers.7.self_attn.k_proj": 8,
+    "thinker.model.layers.7.self_attn.o_proj": 8,
+    "thinker.model.layers.7.self_attn.q_proj": 8,
+    "thinker.model.layers.7.self_attn.v_proj": 8,
+    "thinker.model.layers.8.mlp.down_proj": 8,
+    "thinker.model.layers.8.mlp.gate_proj": 8,
+    "thinker.model.layers.8.mlp.up_proj": 8,
+    "thinker.model.layers.8.self_attn.k_proj": 8,
+    "thinker.model.layers.8.self_attn.o_proj": 8,
+    "thinker.model.layers.8.self_attn.q_proj": 8,
+    "thinker.model.layers.8.self_attn.v_proj": 8,
+    "thinker.model.layers.9.mlp.down_proj": 8,
+    "thinker.model.layers.9.mlp.gate_proj": 8,
+    "thinker.model.layers.9.mlp.up_proj": 8,
+    "thinker.model.layers.9.self_attn.k_proj": 8,
+    "thinker.model.layers.9.self_attn.o_proj": 8,
+    "thinker.model.layers.9.self_attn.q_proj": 8,
+    "thinker.model.layers.9.self_attn.v_proj": 8
+  },
+  "target_modules": ".*",
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

mega-asr-merged/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7fd88dc65e59d502689323b5a296d82f3d8225a38232773963a829e387d44a31
+size 92538896

mega-asr-merged/mega_lora_blocks.json

@@ -0,0 +1,5010 @@
+{
+  "thinker.audio_tower.conv_out": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.0.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.0.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.0.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.0.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.0.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.0.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.1.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.1.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.1.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.1.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.1.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.1.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.10.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.10.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.10.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.10.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.10.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.10.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.11.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.11.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.11.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.11.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.11.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.11.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.12.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.12.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.12.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.12.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.12.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.12.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.13.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.13.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.13.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.13.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.13.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.13.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.14.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.14.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.14.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.14.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.14.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.14.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.15.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.15.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.15.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.15.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.15.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.15.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.16.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.16.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.16.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.16.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.16.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.16.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.17.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.17.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.17.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.17.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.17.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.17.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.18.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.18.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.18.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.18.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.18.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.18.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.19.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.19.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.19.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.19.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.19.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.19.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.2.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.2.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.2.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.2.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.2.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.2.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.20.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.20.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.20.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.20.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.20.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.20.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.21.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.21.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.21.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.21.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.21.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.21.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.22.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.22.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.22.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.22.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.22.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.22.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.23.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.23.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.23.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.23.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.23.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.23.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    },
+    {
+      "alpha": 8,
+      "end": 24,
+      "rank": 8,
+      "start": 16
+    }
+  ],
+  "thinker.audio_tower.layers.3.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.3.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.3.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.3.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.3.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.3.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.4.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.4.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.4.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.4.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.4.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.4.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.5.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.5.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.5.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.5.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.5.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.5.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.6.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.6.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.6.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.6.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.6.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.6.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.7.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.7.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.7.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.7.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.7.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.7.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.8.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.8.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.8.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.8.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.8.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.8.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.9.fc1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.9.fc2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.audio_tower.layers.9.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.9.self_attn.out_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.9.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.layers.9.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.proj1": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.audio_tower.proj2": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    },
+    {
+      "alpha": 8,
+      "end": 16,
+      "rank": 8,
+      "start": 8
+    }
+  ],
+  "thinker.layers.0.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.0.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.0.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.0.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.0.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.0.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.0.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.1.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.1.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.1.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.1.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.1.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.1.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.1.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.10.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.10.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.10.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.10.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.10.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.10.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.10.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.11.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.11.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.11.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.11.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.11.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.11.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.11.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.12.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.12.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.12.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.12.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.12.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.12.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.12.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.13.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.13.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.13.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.13.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.13.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.13.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.13.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.14.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.14.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.14.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.14.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.14.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.14.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.14.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.15.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.15.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.15.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.15.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.15.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.15.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.15.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.16.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.16.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.16.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.16.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.16.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.16.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.16.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.17.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.17.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.17.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.17.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.17.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.17.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.17.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.18.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.18.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.18.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.18.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.18.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.18.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.18.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.19.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.19.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.19.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.19.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.19.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.19.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.19.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.2.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.2.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.2.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.2.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.2.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.2.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.2.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.20.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.20.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.20.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.20.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.20.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.20.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.20.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.21.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.21.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.21.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.21.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.21.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.21.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.21.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.22.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.22.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.22.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.22.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.22.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.22.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.22.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.23.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.23.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.23.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.23.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.23.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.23.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.23.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.24.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.24.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.24.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.24.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.24.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.24.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.24.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.25.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.25.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.25.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.25.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.25.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.25.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.25.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.26.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.26.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.26.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.26.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.26.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.26.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.26.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.27.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.27.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.27.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.27.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.27.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.27.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.27.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.3.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.3.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.3.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.3.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.3.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.3.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.3.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.4.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.4.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.4.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.4.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.4.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.4.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.4.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.5.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.5.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.5.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.5.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.5.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.5.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.5.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.6.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.6.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.6.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.6.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.6.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.6.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.6.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.7.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.7.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.7.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.7.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.7.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.7.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.7.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.8.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.8.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.8.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.8.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.8.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.8.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.8.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.9.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.9.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.9.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.9.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.9.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.9.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.layers.9.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.0.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.0.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.0.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.0.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.0.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.0.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.0.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.1.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.1.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.1.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.1.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.1.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.1.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.1.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.10.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.10.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.10.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.10.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.10.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.10.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.10.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.11.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.11.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.11.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.11.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.11.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.11.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.11.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.12.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.12.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.12.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.12.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.12.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.12.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.12.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.13.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.13.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.13.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.13.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.13.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.13.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.13.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.14.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.14.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.14.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.14.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.14.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.14.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.14.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.15.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.15.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.15.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.15.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.15.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.15.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.15.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.16.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.16.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.16.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.16.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.16.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.16.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.16.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.17.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.17.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.17.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.17.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.17.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.17.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.17.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.18.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.18.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.18.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.18.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.18.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.18.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.18.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.19.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.19.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.19.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.19.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.19.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.19.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.19.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.2.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.2.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.2.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.2.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.2.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.2.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.2.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.20.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.20.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.20.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.20.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.20.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.20.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.20.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.21.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.21.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.21.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.21.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.21.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.21.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.21.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.22.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.22.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.22.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.22.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.22.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.22.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.22.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.23.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.23.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.23.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.23.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.23.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.23.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.23.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.24.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.24.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.24.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.24.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.24.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.24.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.24.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.25.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.25.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.25.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.25.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.25.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.25.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.25.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.26.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.26.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.26.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.26.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.26.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.26.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.26.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.27.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.27.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.27.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.27.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.27.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.27.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.27.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.3.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.3.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.3.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.3.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.3.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.3.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.3.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.4.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.4.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.4.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.4.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.4.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.4.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.4.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.5.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.5.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.5.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.5.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.5.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.5.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.5.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.6.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.6.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.6.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.6.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.6.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.6.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.6.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.7.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.7.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.7.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.7.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.7.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.7.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.7.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.8.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.8.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.8.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.8.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.8.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.8.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.8.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.9.mlp.down_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.9.mlp.gate_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.9.mlp.up_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.9.self_attn.k_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.9.self_attn.o_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.9.self_attn.q_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ],
+  "thinker.model.layers.9.self_attn.v_proj": [
+    {
+      "alpha": 8,
+      "end": 8,
+      "rank": 8,
+      "start": 0
+    }
+  ]
+}