Duplicate from mistralai/Voxtral-4B-TTS-2603

Co-authored-by: Patrick von Platen <patrickvonplaten@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tekken.json filter=lfs diff=lfs merge=lfs -text
+Voxtral_TTS.pdf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,169 @@
+---
+library_name: vllm
+language:
+- en
+- fr
+- es
+- pt
+- it
+- nl
+- de
+- ar
+- hi
+license: cc-by-nc-4.0
+inference: false
+base_model:
+- mistralai/Ministral-3-3B-Base-2512
+extra_gated_description: >-
+  If you want to learn more about how we process your personal data, please read
+  our <a href="https://mistral.ai/terms/">Privacy Policy</a>.
+tags:
+- mistral-common
+pipeline_tag: text-to-speech
+---
+
+# Voxtral 4B TTS 2603 
+
+Voxtral TTS is a frontier, open-weights text-to-speech model that’s fast, instantly adaptable, and produces lifelike speech for voice agents. The model is released with BF16 weights and a set of reference voices. These voices are licensed under CC BY-NC 4, which is the license that the model inherits.
+
+For more details, see our:
+- [🔊 Demo](https://console.mistral.ai/build/audio/text-to-speech)
+- [✍️ Blog post](https://mistral.ai/news/voxtral-tts)
+- [🔬 Research Paper](https://arxiv.org/abs/2603.25551)
+
+
+## Key Features
+
+Voxtral TTS delivers enterprise-grade text-to-speech for production voice agents, with the following capabilities:
+
+- **Realistic, expressive speech** with natural prosody and emotional range across 9 major languages, with support for diverse dialects  
+- **Text-to-Speech generation** with 20 preset voices and easy adaptation to new voices  
+- **Multilingual support**: English, French, Spanish, German, Italian, Portuguese, Dutch, Arabic, and Hindi  
+- **Very low latency** with fast time-to-first-audio, plus streaming and batch inference support  
+- **24 kHz audio output** in WAV, PCM, FLAC, MP3, AAC, and Opus formats  
+- **Production-ready performance** for high-throughput, real-time voice agent workflows
+
+> [!Tip]
+> For voice customization, visit our [AI Studio](https://console.mistral.ai/build/audio/text-to-speech).
+
+### Use Cases
+
+- Customer support and call center infrastructure.
+- Financial services. _-- with video demo on banking KYC voice agents._
+- Manufacturing and industrial operations.
+- Public services and government.
+- Compliance and risk.
+- Supply chain and logistics.
+- Automotive and in-vehicle systems.
+- Sales and marketing.
+- Real-time translation.
+
+> [!Warning]
+> Responsible Use - 
+> You are responsible for complying with applicable laws and avoiding misuse.
+
+## Benchmark Results
+
+  - Measured using [vllm_omni/examples/offline_inference/voxtral_tts/end2end.py](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/voxtral_tts).
+  - Input: 500-character text with a 10-second audio reference.
+  - Hardware: single NVIDIA H200.
+  - vllm version: v0.18.0.
+
+*Note*: The RTF in `end2end.py` uses an inverted formula (higher = better). The table below converts it back to the standard RTF convention (lower = better)
+
+  | Concurrency | Latency | RTF   | Throughput (char/s/GPU) |
+  |:-----------:|:-------:|:-----:|:-----------------------:|
+  | 1           | 70 ms   | 0.103 | 119.14                  |
+  | 16          | 331 ms  | 0.237 | 879.11                  |
+  | 32          | 552 ms  | 0.302 | 1430.78                 |
+
+
+## Usage
+
+The model can also be deployed with the following libraries:
+- [`vllm-omni (recommended)`](https://github.com/vllm-project/vllm-omni): See [here](#vllm-omni-recommended)
+
+### vLLM Omni (recommended)
+
+> [!Tip]
+> We've worked hand-in-hand with the vLLM-Omni team to have production-grade support for Voxtral 4B TTS 2603 with vLLM-Omni.
+> Special thanks goes out to Han Gao, Hongsheng Liu, Roger Wang, and Yueqian Lin from the vLLM-Omni team.
+
+
+**Installation**
+
+Make sure to install [vllm](https://github.com/vllm-project/vllm) from the latest (>= 0.18.0) pypi package. 
+See [here](https://docs.vllm.ai/en/latest/getting_started/installation/) for a full installation guide.
+
+```
+uv pip install -U vllm
+```
+
+Next, you should install [`vllm-omni`](https://github.com/vllm-project/vllm-omni) with `vllm-omni >= 0.18.0`.
+
+```
+uv pip install vllm-omni --upgrade  # make sure to have >= 0.18.0
+```
+
+Alternatively, you can also make use of a ready-to-go docker image on the [docker hub](https://hub.docker.com/layers/vllm/vllm-omni/v0.18.0/images/sha256-d855c9f3e06b1126e8a082229e5d2fef217e43c98d03569f8b9e50fa5c2d0a61).
+
+
+Installing `vllm >= 0.18.0` should automatically install `mistral_common >= 1.10.0` which you can verify by running:
+
+```sh
+python3 -c "import mistral_common; print(mistral_common.__version__)" # should print >= 1.10.0
+```
+
+#### Serve
+
+Due to size and the BF16 format of the weights - `Voxtral-4B-TTS-2603` can run on a single GPU with >= 16GB memory.
+
+```bash
+vllm serve mistralai/Voxtral-4B-TTS-2603 --omni
+```
+
+#### Client
+
+```py
+import io
+import httpx
+import soundfile as sf
+ 
+BASE_URL = "http://<your-server-url>:8000/v1"
+ 
+payload = {
+    "input": "Paris is a beautiful city!",
+    "model": "mistralai/Voxtral-4B-TTS-2603",
+    "response_format": "wav",
+    "voice": "casual_male",
+}
+ 
+response = httpx.post(f"{BASE_URL}/audio/speech", json=payload, timeout=120.0)
+response.raise_for_status()
+ 
+audio_array, sr = sf.read(io.BytesIO(response.content), dtype="float32")
+print(f"Got audio: {len(audio_array)} samples at {sr} Hz")
+
+# you can play the audio with a library like `sounddevice.play` for example
+```
+
+#### Demo
+
+To run it:
+
+```sh
+git clone https://github.com/vllm-project/vllm-omni.git && \
+cd vllm-omni && \
+uv pip install gradio==5.50 && \
+python examples/online_serving/voxtral_tts/gradio_demo.py \
+  --host <your-server-url> \
+  --port 8000
+```
+
+Alternatively you can also try it out live here ➡️ [**HF Space**](https://huggingface.co/spaces/mistralai/voxtral-tts-demo).
+
+## License
+
+The provided voice-references compatible with this model are licensed under [CC BY-NC 4](https://creativecommons.org/licenses/by-nc/4.0/), e.g. from EARS, CML-TTS, IndicVoices-R and Arabic Natural Audio datasets. Thus, this model inherits the same license.
+
+*You must not use this model in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.*

consolidated.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66c4fd998db10e1a6d9cc5baa10e6264bf10701ec22ccdc0822c7dcc45dbe55b
+size 8004752248

params.json

@@ -0,0 +1,130 @@
+{
+  "dim": 3072,
+  "n_layers": 26,
+  "head_dim": 128,
+  "hidden_dim": 9216,
+  "n_heads": 32,
+  "n_kv_heads": 8,
+  "fp8_matmul": false,
+  "use_biases": false,
+  "causal": true,
+  "rope_theta": 1000000.0,
+  "norm_eps": 1e-05,
+  "init": "NO_INIT",
+  "dropout": 0.0,
+  "vocab_size": 131072,
+  "model_parallel": 1,
+  "is_sequence_parallel": false,
+  "context_parallel": 1,
+  "tied_embeddings": true,
+  "shard_on_vocab_dim": false,
+  "model_pipelining": 1,
+  "virtual_model_pipelining": 1,
+  "fused_rms_norm": true,
+  "checkpoint": false,
+  "use_cache": false,
+  "max_concurrent_tokens": 65536,
+  "learnable_sinks": false,
+  "rms_norm": "PRE",
+  "cust_bwd": false,
+  "recompute_w1_every": 0,
+  "recompute_w3_every": 0,
+  "recompute_attn_every": 0,
+  "freeze_nonembedding": false,
+  "fsdp2": true,
+  "dp_replicate_size": 1,
+  "zero2": true,
+  "fsdp_optimize_backward_concat_if_pp": true,
+  "attention_type": "FLASH_ATTN_3",
+  "multimodal": {
+    "bos_token_id": 1,
+    "audio_model_args": {
+      "semantic_codebook_size": 8192,
+      "acoustic_codebook_size": 21,
+      "n_acoustic_codebook": 36,
+      "audio_encoding_args": {
+        "codebook_pattern": "parallel",
+        "interleave_audio_tokens_per_segment": 8192,
+        "interleave_text_tokens_per_segment": 8192,
+        "single_trailing_segment": false,
+        "num_codebooks": 37,
+        "sampling_rate": 24000,
+        "frame_rate": 12.5
+      },
+      "audio_token_id": 24,
+      "begin_audio_token_id": 25,
+      "input_embedding_concat_type": "sum",
+      "acoustic_transformer_args": {
+        "input_dim": 3072,
+        "dim": 3072,
+        "n_layers": 3,
+        "head_dim": 128,
+        "hidden_dim": 9216,
+        "n_heads": 32,
+        "n_kv_heads": 8,
+        "use_biases": false,
+        "rope_theta": 10000.0,
+        "sigma": 1e-05,
+        "sigma_max": 1.0
+      },
+      "p_uncond": 0.0,
+      "text_feature_bugged": false,
+      "condition_dropped_token_id": 42
+    },
+    "audio_tokenizer_args": {
+      "channels": 1,
+      "sampling_rate": 24000,
+      "pretransform_patch_size": 240,
+      "patch_proj_kernel_size": 7,
+      "semantic_codebook_size": 8192,
+      "semantic_dim": 256,
+      "acoustic_codebook_size": 21,
+      "acoustic_dim": 36,
+      "conv_weight_norm": true,
+      "causal": true,
+      "attn_sliding_window_size": 16,
+      "half_attn_window_upon_downsampling": true,
+      "dim": 1024,
+      "hidden_dim": 4096,
+      "head_dim": 128,
+      "n_heads": 8,
+      "n_kv_heads": 8,
+      "qk_norm_eps": 1e-06,
+      "qk_norm": true,
+      "use_biases": false,
+      "norm_eps": 0.01,
+      "layer_scale": true,
+      "layer_scale_init": 0.01,
+      "decoder_transformer_lengths_str": "2,2,2,2",
+      "decoder_convs_kernels_str": "3,4,4,4",
+      "decoder_convs_strides_str": "1,2,2,2",
+      "voice": {
+        "casual_female": 0,
+        "casual_male": 1,
+        "cheerful_female": 2,
+        "neutral_female": 3,
+        "neutral_male": 4,
+        "pt_male": 5,
+        "pt_female": 6,
+        "nl_male": 7,
+        "nl_female": 8,
+        "it_male": 9,
+        "it_female": 10,
+        "fr_male": 11,
+        "fr_female": 12,
+        "es_male": 13,
+        "es_female": 14,
+        "de_male": 15,
+        "de_female": 16,
+        "ar_male": 17,
+        "hi_male": 18,
+        "hi_female": 19
+      }
+    }
+  },
+  "torch_compile_swiglu_noncust_bwd": false,
+  "override_parameters_str": "",
+  "max_seq_len": 65536,
+  "model_type": "voxtral_tts",
+  "max_position_embeddings": 128000
+}

tekken.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:587989c9f56676b35e7d16d6fc61461301e402d908392a8ce16f0349f61b56d7
+size 14894731

voice_embedding/ar_male.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f44603f6433cbb4b2abc7f496a382632171118557a175cb385df168a0dc20464
+size 413253

voice_embedding/casual_female.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:780637984644064ee22e60b3152e0cd43fa64b2dcd39d9cab6cd2c62f2ce0342
+size 1316421

voice_embedding/casual_male.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a056c9156ad0058e9d1368363bf3a25a9fcd8fe53e211ffac97de0bbffb3504
+size 904773

voice_embedding/cheerful_female.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:75fe69c8fcb5a0883a3d0bc1215b28f28cc0586aff5732eeebd2b254e8288253
+size 812613

voice_embedding/de_female.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:282fc191fda496de2ebf2c809acb44056dde6fbe2f1cb99e85e67985bc6f6619
+size 904773

voice_embedding/de_male.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd75d9fd3ffb9df0481668ce8781287a58f552e2388c5bbc0efdd4ebff0421bf
+size 1003077

voice_embedding/es_female.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90e01ad34f231cc881987c3b1c0728853fd9b904e52c296a07c71a132949d8a6
+size 849477

voice_embedding/es_male.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec116d8f4a102291bae3d9156d7c3222d9e1056020bf5894a7504bfc09640fdf
+size 1279557

voice_embedding/fr_female.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82628d963670f919aa302f9c8a7336c745418a145934edb211810b07d9c8b852
+size 597573

voice_embedding/fr_male.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:73395073472be3fb586b487705ac4ebf35f99db664f56400137e8bfcfe4cd8a8
+size 597573

voice_embedding/hi_female.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa7718cdd6f65735226bcc701379fdec64f36d0207ca79fc4c61b445ca7bde82
+size 529989

voice_embedding/hi_male.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3cde36ab9a336f67fd33b46435cdf645cff9e10117f13bcbcb67b44b80a11b0
+size 579141

voice_embedding/it_female.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29e1714bdb3ce0726e590ce1862fbe953c168ba51a05bc7daa8cb35cddc312b4
+size 1058373

voice_embedding/it_male.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b98ba2253e2a0b872e20d33d29cab32263cc81062c01e3f5a8696de89e6f47b1
+size 1033797

voice_embedding/neutral_female.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a03f4008614da7b1505a360a6b0d58d94dd72b0b0f49bf216e39de5eb733c61
+size 1340997

voice_embedding/neutral_male.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:439df812990e6e4bcc6010ca12f12df90916e862bc1e1b56036d6433b892834e
+size 1039941

voice_embedding/nl_female.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1bad34c22e0563f05c1f13c1db96680778c297aea6a5c0bb202950648b796b6
+size 898629

voice_embedding/nl_male.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43fd2de89dc08503f37ae3107273eeb3f2a6195d705ff58d2228b3b5642ff7de
+size 849477

voice_embedding/pt_female.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82f1006b2cd69118cba67085daa1795d9dab90b9bc70e1392e77f82cb616c9ce
+size 1076805

voice_embedding/pt_male.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7b30dca6c5d16c7b10a1c09c53e971c1bb1fab65692d7244876fbdc4ad52ba18
+size 886341