Initial release: Dramabox v1 - Expressive TTS with Voice Cloning

.gitattributes

@@ -0,0 +1,43 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+samples/01_queen_sighs_rage.wav filter=lfs diff=lfs merge=lfs -text
+samples/refs/01_queen_sighs_rage.wav filter=lfs diff=lfs merge=lfs -text
+samples/04_catgirl_giggles_snort.wav filter=lfs diff=lfs merge=lfs -text
+samples/refs/04_catgirl_giggles_snort.wav filter=lfs diff=lfs merge=lfs -text
+samples/06_arnie_panting_triumph.wav filter=lfs diff=lfs merge=lfs -text
+samples/09_villain_sinister_laugh.wav filter=lfs diff=lfs merge=lfs -text
+samples/refs/09_villain_sinister_laugh.wav filter=lfs diff=lfs merge=lfs -text
+samples/13_conan_wheezing_laughter.wav filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,159 @@
+---
+language:
+  - en
+  - hi
+  - es
+  - de
+  - fr
+  - ja
+  - it
+  - ko
+  - pt
+  - zh
+license: other
+pipeline_tag: text-to-speech
+tags:
+  - tts
+  - voice-cloning
+  - audio-generation
+  - diffusion-transformer
+  - flow-matching
+  - ltx-2
+library_name: ltx-audio-tts
+---
+
+# Dramabox - Expressive TTS with Voice Cloning
+
+Dramabox generates expressive, emotionally rich speech from scene descriptions with optional voice cloning. Built on a 3.3B Diffusion Transformer with flow matching, conditioned on Gemma 3 12B text embeddings.
+
+## Audio Samples
+
+### Regal Queen - Cold Fury to Venomous Whisper
+
+**Prompt:** A regal woman speaks with cold fury in a measured, low voice. She sighs deeply, "I have told you a thousand times, and yet here we are again." Her voice sharpens with rising anger, "Do you honestly think I enjoy repeating myself?! Do you?!" She lets out a cold, mocking laugh, "Hahaha, how utterly pathetic you are." She drops to a venomous whisper, leaning close, "Now get out of my sight before I do something we will both regret."
+
+<audio controls src="https://huggingface.co/ResembleAI/Dramabox/resolve/main/samples/01_queen_sighs_rage.wav"></audio>
+
+### Catgirl - Uncontrollable Giggling
+
+**Prompt:** A playful girl speaks in a bright, singsong voice, already mid-giggle, "Hehehe, oh my gosh you should see your face right now, it is priceless!" She gasps for air between giggles, "Oh my, hehe, oh my, I cannot stop laughing!" She tries to compose herself with a long sigh, "Ahhhhh okay okay okay, I will stop, I promise I will stop." She leans in and whispers conspiratorially, "But seriously though, between you and me," then immediately loses it again, "Haha, no I, hehehe, I just cannot! You are way too funny, haha!" She snorts mid-laugh, "Pfft, oh no no no, that was so embarrassing, pretend you did not hear that!"
+
+<audio controls src="https://huggingface.co/ResembleAI/Dramabox/resolve/main/samples/04_catgirl_giggles_snort.wav"></audio>
+
+### Action Hero - Panting Triumph
+
+**Prompt:** A muscular man speaks with a thick accent, panting heavily, completely out of breath, "Hah... hah... we made it, we actually made it." He coughs roughly, "Ugh, that was the hardest fight of my entire life, I swear." He groans and clutches his side, "Argh, my ribs, I think something is broken." But then a grin spreads and he laughs heartily despite the pain, "Hahaha! But we WON! Can you believe it? We actually won!" He takes a deep, shuddering breath, "I told you, heh, I told you we would make it. Ahhh, it is finally over."
+
+<audio controls src="https://huggingface.co/ResembleAI/Dramabox/resolve/main/samples/06_arnie_panting_triumph.wav"></audio>
+
+### Villain - Sinister Laugh
+
+**Prompt:** A deep-voiced villain speaks with theatrical menace, chuckling softly at first, "Heheheh. Hahahahahahaha! Oh, forgive me, forgive me." He catches his breath with a sinister grin, He clears his throat. "It is just SO amusing when they struggle, is it not?" His voice drips with contempt, "I expected more from you, truly I did. How disappointing." He leans in close and whispers with vicious intensity, "But fear not, my dear. The REAL entertainment has only just begun." He chuckles one last time, "Heheheh."
+
+<audio controls src="https://huggingface.co/ResembleAI/Dramabox/resolve/main/samples/09_villain_sinister_laugh.wav"></audio>
+
+### Talk Show Host - Wheezing Laughter
+
+**Prompt:** A talk show host speaks with animated enthusiasm. He gasps with exaggerated shock, "No! You did NOT just say that, tell me you did not just say that!" He bursts into uncontrollable laughter, "HAHAHA! Oh my god, oh my god!" He wheezes, barely getting words out, "I cannot, I literally cannot breathe right now!" He wipes his eyes, sniffling, "Oh that is so good, that is really genuinely good." He sighs happily, "Ahhh okay okay, let me compose myself, I am a professional." He takes one breath then immediately cracks up again, "Pfft hehehe, no I absolutely cannot, I am so sorry everybody!" He claps, "Folks, THIS, this right here, is why I love my job!"
+
+<audio controls src="https://huggingface.co/ResembleAI/Dramabox/resolve/main/samples/13_conan_wheezing_laughter.wav"></audio>
+
+---
+
+## Model Description
+
+Dramabox is a prompt-driven TTS model where **the text prompt controls everything** - speaker identity, emotion, delivery style, laughs, sighs, pauses, and transitions. With voice cloning, a 10-second reference clip conditions the model to reproduce the speaker's timbre and characteristics.
+
+### Key Features
+
+- **Prompt-driven expressiveness** - laughs, sighs, whispers, shouts, emotional transitions all controlled by the scene description
+- **Voice cloning** from 10s reference audio
+- **10 languages** - EN, HI, ES, DE, FR, JA, IT, KO, PT, ZH
+- **Fast inference** - ~2.5s per generation with warm server on H100
+
+### Architecture
+
+| Component | Details |
+|-----------|---------|
+| **Transformer** | 3.3B parameter DiT, 48 layers, flow matching (30-step Euler) |
+| **Text Encoder** | Gemma 3 12B (q4 quantized) + learned embeddings processor |
+| **Audio VAE** | Encodes/decodes 48kHz audio via mel spectrogram latents |
+| **Voice Cloning** | Reference audio tokens appended to target with asymmetric attention mask |
+
+## Files
+
+| File | Size | Description |
+|------|------|-------------|
+| `dramabox-dit-v1.safetensors` | 6.6 GB | DiT transformer (voice cloning weights merged) |
+| `dramabox-audio-components.safetensors` | 2.7 GB | Audio VAE encoder/decoder + vocoder + text projection |
+| `assets/silence_latent_frame.pt` | 1.5 KB | VAE-encoded silence frame |
+| `config.json` | - | Model configuration |
+
+**Additional requirement**: [unsloth/gemma-3-12b-it-bnb-4bit](https://huggingface.co/unsloth/gemma-3-12b-it-bnb-4bit) (text encoder, pre-quantized 4-bit, auto-downloaded)
+
+## Quick Start
+
+```python
+from inference_server import TTSServer
+
+# Models auto-download from HuggingFace
+server = TTSServer(device="cuda", bnb_4bit=True)
+
+# Text-to-speech
+server.generate_to_file(
+    prompt='A woman speaks warmly, "Hello, how are you today?" She laughs, "Hahaha, it is so good to see you!"',
+    output="output.wav",
+)
+
+# Voice cloning
+server.generate_to_file(
+    prompt='A woman speaks warmly, "Hello, how are you today?"',
+    output="cloned.wav",
+    voice_ref="reference.wav",  # 10+ seconds of target voice
+)
+```
+
+## Prompt Format
+
+The prompt is a scene description that controls how the model speaks:
+
+```
+<speaker description>, "<dialogue>" <action direction> "<more dialogue>"
+```
+
+### What Works Inside Quotes (model produces actual sounds)
+- Laughs: `"Hahaha"` `"Hehehe"` (always as one word, never separated)
+- Sounds: `"Mmmmm"` `"Ugh"` `"Argh"` `"Ahhh"` `"Hmm"`
+
+### What Goes Outside Quotes (stage directions)
+- `She sighs deeply.` `He gulps nervously.` `A long pause.`
+- `Her voice cracks.` `He clears his throat.` `She scoffs.`
+
+### Never Inside Quotes (model speaks them literally)
+- Ahem, Pfft, Sigh, Gasp, Cough
+
+## Inference Settings
+
+| Parameter | Default | Notes |
+|-----------|---------|-------|
+| cfg_scale | 2.5 | Text adherence (lower = more natural) |
+| stg_scale | 1.5 | Skip-token guidance |
+| rescale | 0.0 | No rescaling |
+| modality | 1.0 | No modality guidance |
+| duration_multiplier | 1.1 | 10% extra breathing room |
+| steps | 30 | Euler flow matching |
+
+## VRAM Requirements
+
+| Setup | VRAM | Speed |
+|-------|------|-------|
+| Warm server (recommended) | **~24 GB** | **~2.5s** |
+| Cold inference (per-call loading) | ~8 GB peak | ~30s |
+
+## Supported Languages
+
+English, Hindi, Spanish, German, French, Japanese, Italian, Korean, Portuguese, Mandarin
+
+## License
+
+Built on [LTX-2.3](https://github.com/Lightricks/LTX-2) by Lightricks. Please refer to the LTX-2 license for usage terms.

assets/silence_latent_frame.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f73746d2163f8f1742c5de89005404ccaeeff05154bbb10a3337bf9bd13f161c
+size 1501

config.json

@@ -0,0 +1,42 @@
+{
+  "model_type": "dramabox-tts",
+  "architecture": "DiT-FlowMatching",
+  "base_model": "ltx-2.3-22b-dev-audio-only",
+  "parameters": "3.3B",
+  "num_layers": 48,
+  "audio_inner_dim": 2048,
+  "audio_num_attention_heads": 32,
+  "audio_attention_head_dim": 64,
+  "audio_cross_attention_dim": 2048,
+  "denoising_steps": 30,
+  "scheduler": "euler_flow_matching",
+  "text_encoder": "google/gemma-3-12b-it-qat-q4_0-unquantized",
+  "text_encoder_hidden_size": 3840,
+  "ic_lora": {
+    "rank": 128,
+    "alpha": 128,
+    "merged": true,
+    "training_version": "v13",
+    "text_dropout": 0.4,
+    "training_steps": "v12@3000 + v13@1000"
+  },
+  "audio": {
+    "sample_rate": 48000,
+    "vae_channels": 8,
+    "mel_bins": 16,
+    "fps": 25.0
+  },
+  "inference_defaults": {
+    "cfg_scale": 2.5,
+    "stg_scale": 1.5,
+    "rescale_scale": 0.0,
+    "modality_scale": 1.0,
+    "duration_multiplier": 1.1,
+    "seed": 42
+  },
+  "files": {
+    "transformer": "dramabox-dit-v1.safetensors",
+    "audio_components": "dramabox-audio-components.safetensors",
+    "silence_latent": "assets/silence_latent_frame.pt"
+  }
+}

dramabox-audio-components.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6bb7195f91ffac65f8773215851bf751c86bab9f7d130e9fc29e9fef2bd7954
+size 2676984708

dramabox-dit-v1.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:248d292627f8fa67ed3e587171c28051edb2c06ce7d2d2a9e15132f0bff0540f
+size 6573055336

samples/01_queen_sighs_rage.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:758fb1412f9af73721e59a6e4c949bbd14aeda802d52786471fe3130a84a447e
+size 4855758

samples/04_catgirl_giggles_snort.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d12ac377fceed7488493e6ee4ae9c8d7f9294bb64822068fc54e2e6350ca1453
+size 7620558

samples/06_arnie_panting_triumph.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68b67153ba93c500254f963840b46da1439785e50b3ff432df8f9d8c3f47a035
+size 6268878

samples/09_villain_sinister_laugh.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd2368c3fad976f9ce54cf8d0608a78574ab83a7584d5754c3703c9ade64fb69
+size 5285838

samples/13_conan_wheezing_laughter.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8ee6e28c11c7844599213f8eebe72ab20dc43c55621ca453e16cad0609d45d3
+size 7190478

samples/refs/01_queen_sighs_rage.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c0bf624251cc325098863e3b5e280505c4dccfd5591e6312a1844a467b1a3f14
+size 351616

samples/refs/04_catgirl_giggles_snort.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6e4a21b962c30a2644a6e7f6b5e2b0a7db8b63d2cf2efa69b009bd9b62b0bf3
+size 414478

samples/refs/09_villain_sinister_laugh.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41f266980881a7c61027f73831b559dde846469e74966d37bb06c52992ae472c
+size 349946