| --- |
| license: openrail |
| language: |
| - en |
| - ja |
| - zh |
| - ko |
| - es |
| - fr |
| - de |
| - multilingual |
| library_name: coremltools |
| tags: |
| - coreml |
| - ane |
| - apple-neural-engine |
| - text-to-speech |
| - tts |
| - audio |
| - diffusion |
| - flow-matching |
| - on-device |
| - ios |
| - macos |
| - fp16 |
| pipeline_tag: text-to-speech |
| base_model: Supertone/supertonic-3 |
| base_model_relation: quantized |
| --- |
| |
| # Supertonic-3 — CoreML (fp16, ANE-ready) |
|
|
| CoreML conversion of [Supertone/supertonic-3](https://huggingface.co/Supertone/supertonic-3), |
| a 99M-parameter multilingual TTS model. All 4 components run on the |
| Apple Neural Engine (1.8–3.7× faster than CPU on M-series chips). |
|
|
| | Component | Size | Role | |
| | --- | ---: | --- | |
| | `fp16/duration_predictor.mlpackage` | 15 MB | text -> frame count | |
| | `fp16/text_encoder.mlpackage` | 71 MB | text -> conditioning latent | |
| | `fp16/vector_estimator.mlpackage` | 135 MB | flow-matching denoiser (8 steps) | |
| | `fp16/vocoder.mlpackage` | 51 MB | latent -> 44.1 kHz waveform | |
| | **Total** | **272 MB** | (originals: ~400 MB ONNX) | |
|
|
| ## Quickstart |
|
|
| ```bash |
| pip install coremltools soundfile numpy supertonic |
| git clone https://huggingface.co/Reza2kn/supertonic-3-coreml |
| cd supertonic-3-coreml |
| |
| # Short prompt |
| python inference.py --text "Hello, world." --voice F1 --lang en --out hello.wav |
| |
| # Long prompt — use --auto-pad for full content rendering |
| python inference.py \ |
| --text "A gentle breeze moved through the open window while everyone listened to the story. The narrator paused, took a slow breath, and continued in a softer tone. Outside, the city carried on, unaware of the quiet moment unfolding inside." \ |
| --voice F5 --lang en --auto-pad --out long.wav |
| ``` |
|
|
| 10 voice styles ship in `voice_styles/`: F1–F5 (female), M1–M5 (male). |
| 31 languages supported via `unicode_indexer.json`. |
|
|
| ## The auto-pad trick (why `--auto-pad` matters) |
|
|
| The supertonic-3 model has a soft cap on how much speech it renders per |
| utterance. For long inputs (more than ~13 s of natural speech) the model |
| truncates the prompt and emits a low-amplitude filler tone for the rest |
| of the budget. The CoreML conversion's static bucket (T=L=320) extends |
| this cap by ~3 s due to the way the bucket's padded positions leak into |
| the real positions through ConvNeXt's dilated convolutions — that's |
| **why CoreML inference sounds more natural than the original ONNX |
| library** (proper word separation, intonation), but it still cuts off |
| mid-sentence on long prompts. |
|
|
| `--auto-pad` is a two-pass workaround: |
|
|
| 1. **Pass 1** synthesizes the prompt alone at full bucket length to find |
| where the model's content naturally stops (`t_orig`). |
| 2. **Pass 2** appends a long filler sentence |
| (`" And with that, the gentle silence wrapped itself around the room."`) |
| that gives the model extra frames to fully render the original |
| prompt, then renders the filler sentence, then drops into the filler |
| tone. |
| 3. The longest clean-silence gap after `t_orig` is the boundary between |
| the original prompt and the appended filler. The pipeline trims |
| there and tail-pads with 0.5 s of true silence. |
|
|
| Cost: ~2× synthesis time. Worth it for any prompt over ~5 s. |
|
|
| ## ANE engagement |
|
|
| All 4 components compile to ANE-resident programs when loaded with |
| `compute_units=ALL` (default). Measured speedups on M2 Pro vs CPU: |
|
|
| | Component | ANE speedup | |
| | --- | --- | |
| | duration_predictor | 1.9× | |
| | text_encoder | 2.8× | |
| | vector_estimator | 2.4× (per step; 8 steps total) | |
| | vocoder | 3.7× | |
| |
| Verify ANE engagement with: |
| |
| ```bash |
| xctrace record --template "Core ML" --output trace.trace -- python inference.py --text "test" |
| ``` |
| |
| ## Conversion notes |
| |
| - Static bucket: T=320 (text length), L=320 (latent length). Inputs are |
| zero-padded on the right and masked. Bucket = 22.3 s of audio. |
| - `duration_predictor`, `text_encoder`, `vocoder` are hand-reimplemented |
| in PyTorch from the ONNX initializers, then traced to CoreML. |
| Per-component float parity vs ONNX: max-abs 2e-5 (dp), cos 0.998 |
| (text_encoder), cos 0.9998 (vocoder). |
| - `vector_estimator` (the heavy diffusion model) goes through |
| `onnxsim.simplify(T=L=320)` -> `onnx2torch.convert` -> `torch.jit.trace` |
| -> coremltools. Cos 0.998 vs ONNX per diffusion step. |
| - The diffusion sampler stays host-side (8 Euler steps over the single |
| step graph). All 4 components are individually quantizable. |
|
|
| ## License |
|
|
| This conversion follows the original Supertone/supertonic-3 license |
| (OpenRAIL). See `LICENSE` (or the upstream model card). |
|
|
| ## Why fp16 and not INT4? |
|
|
| We attempted to ship an INT4 variant. After exhaustive testing |
| ([INT4 sweep notes below](#int4-sweep-results)), the supertonic-3 |
| architecture caps at INT8 minimum for the vocoder and vector_estimator: |
| |
| | Component | INT4 result | Why | |
| | --- | --- | --- | |
| | **vocoder** | cos ≈ 0 across all 6 palette configs (kmeans/uniform, pgc with group_size 8/16/32/64). Mixed-precision (head-only carve-out) also broken. | HiFi-GAN-style upsampling is uniformly sensitive — INT8 (cos 0.99) is the floor. | |
| | **vector_estimator** | per-step cos 0.82-0.999; over 8 diffusion steps this compounds to garbled / "backwards-sounding" audio. | Flow-matching diffusion accumulates error. INT8 works (per-step cos 0.96-0.998 except short_ja_F3 which is architecturally cos 0.82 even at INT8). | |
| | **duration_predictor** | smallest drift was 0.11s with pt_uniform — but enough to shift L_real → bucket-leak boundary moves → pacing perceptibly breaks. | dp output sets the diffusion frame budget; any drift propagates. | |
| | **text_encoder** | cos 0.97 at pgc_g32 (works alone). | Conditioning quality compounds with VE drift. | |
| |
| The best achievable mixed config (only voc INT8, others fp16) saves |
| ~25 MB out of 272 MB — not worth a separate variant. The fp16 build |
| shipped here is the final deliverable. |
| |
| ## Companion build |
| |
| The cross-platform LiteRT version is at |
| [Reza2kn/supertonic-3-litert](https://huggingface.co/Reza2kn/supertonic-3-litert). |
| LiteRT ships INT4/INT8 + an INT8 quantized ONNX vector_estimator |
| (65 MB instead of 256 MB) — but LiteRT can't reproduce the |
| CoreML-only "bucket-leak" extension, so long prompts sound rushed on |
| LiteRT. Use CoreML for full quality on Apple platforms. |
| |
| ## Credits |
| |
| - Original model: [Supertone/supertonic-3](https://huggingface.co/Supertone/supertonic-3) |
| - CoreML conversion + auto-pad workflow: this repo |
| - Companion LiteRT build: [Reza2kn/supertonic-3-litert](https://huggingface.co/Reza2kn/supertonic-3-litert) |
| |
