--- license: apache-2.0 language: - en - zh - ja - ko - multilingual library_name: coremltools tags: - coreml - ane - apple-neural-engine - automatic-speech-recognition - asr - speech-recognition - robust-asr - quantized - int4 - 4bit - 8bit - mixed-precision - lut - palettize - on-device - apple-silicon - ios - macos - qwen3 - qwen3-asr - mega-asr - anemll pipeline_tag: automatic-speech-recognition base_model: zhifeixie/Mega-ASR base_model_relation: quantized --- # Mega-ASR — CoreML mixed 8/4 (end-to-end ASR) CoreML deployment of [zhifeixie/Mega-ASR](https://huggingface.co/zhifeixie/Mega-ASR) (Qwen3-ASR-1.7B base) with an **`input_embeds`-aware decoder** so audio embeddings can be scattered at `<|audio_pad|>` positions to do real ASR — not just text generation. Converted via [ANEMLL](https://github.com/Anemll/Anemll) with a custom `convert_embeds_mixed.py` that: 1. Monkey-patches `QwenModel.forward` + `QwenForCausalLM.forward` to accept pre-embedded `hidden_states` (skipping the internal `embed_tokens` lookup) so audio scatter works at inference. 2. Enumerates the MIL program's const-weight ops by name pattern and applies **LUT-8 palettization to attention projections** (q/k/v/o_proj) and **LUT-4 to MLP projections** (gate/up/down_proj) — mirroring the MLX `mixed8_4` recipe that closed the gap to GPTQ on the LLM portion. 3. Runs `compute_precision=FLOAT32` — fp16 compute precision produces all-NaN logits on Qwen3-ASR's RMSNorm/attention (matches the aoiandroid community finding for the same base model). ## What's in this repo | File | Size | Role | | --- | ---: | --- | | `coreml/mega-asr-llm-embeds_mixed8_4.mlpackage/` | **1.87 GB** | **Recommended.** Qwen3 1.7B LLM, `inputs_embeds` input, fp32 compute, 8-bit attn + 4-bit MLP, ~5.0 bpw avg. | | `coreml/mega-asr-llm-embeds_fp32compute_lut4.mlpackage/` | 826 MB | Smaller variant. Uniform LUT-4 weights. -3.7% agreement vs mixed. | | `coreml/mega-asr-llm_lut4.mlpackage/` | 974 MB | Standalone Qwen3 text LLM with `input_ids` input (no audio scatter). | | `onnx/audio_encoder_fp32.onnx` | 1.27 GB | 24-layer Whisper-style audio encoder (ONNX, runs via onnxruntime; CoreML port pending) | | `tokenizer/*` | — | Original Qwen3-ASR tokenizer (`<\|audio_pad\|>`, ``, etc.) | | `examples/*.wav` | ~3 MB | 8 noisy benchmark clips from Voices-in-the-Wild-Bench | | `inference_asr.py` | — | End-to-end ASR pipeline (ONNX encoder + CoreML LLM) | | `convert_embeds.py` / `convert_embeds_mixed.py` | — | The custom converters | ## Quality (bench) 8-clip [Voices-in-the-Wild-Bench](https://github.com/xzf-thu/Voices-in-the-Wild-Bench) agreement (1 − WER), prompt forced to `language English`, ONNX fp32 audio encoder + the CoreML LLM, ran with `compute_units=ALL` (Metal GPU since ANE compilation fails on this model size + stateful KV cache): | Per-sample | Mixed 8/4 (recommended) | Uniform LUT-4 | | --- | ---: | ---: | | distortion | 100% | 100% | | dropout | 100% | 100% | | echo (hard, heavy reverb) | **64.7%** | 47.1% | | far_field | 100% | 100% | | mixed | 100% | 100% | | noise | 100% | 100% | | obstructed | **100%** | 88.2% | | recording (hard, truncated audio) | 60.0% | 60.0% | | **AVERAGE** | **90.6%** | 86.9% | Mixed 8/4 lifts CoreML from 86.9% → 90.6% (+3.7) by allocating the 4 attention projections per layer to LUT-8 (16 unique values for every 8 channels) while keeping the 3 MLP projections at LUT-4 (16 unique values per 8 channels). Attention layers in Qwen3 are quality-critical — same result we found in the MLX port. Cross-backend leaderboard (same 8 samples, same audio encoder): | Backend | Agreement | | --- | ---: | | ONNX recommended (GPTQ INT4) | 92.7% | | MLX recommended (mixed 8/4) | 92.2% | | **CoreML recommended (mixed 8/4)** | **90.6%** | | CoreML LUT-4 baseline | 86.9% | | ONNX RTN INT4 baseline | 87.8% | The remaining ~2% gap to ONNX/MLX is the LUT-vs-GPTQ scheme difference (k-means clustering vs activation-aware Hessian redistribution). The two hard samples (`echo`, `recording`) are audio-quality-limited and stay around 60-65% across all 4-bit backends. ## Inference ```bash pip install coremltools onnxruntime soundfile transformers safetensors librosa numpy git clone https://huggingface.co/Reza2kn/mega-asr-coreml cd mega-asr-coreml python inference_asr.py \ --mlpackage coreml/mega-asr-llm-embeds_mixed8_4.mlpackage \ --encoder-path onnx/audio_encoder_fp32.onnx \ --examples-dir examples \ --qwen-asr-dir \ --compute-unit ALL ``` The pipeline: 1. **Mel features** via Qwen3-ASR's `WhisperFeatureExtractor`. 2. **Audio encoder** (ONNX fp32) → audio embeddings `(F, 2048)`. 3. **Prompt + scatter**: build the Qwen3-ASR chat template with English forcing, expand the single `<|audio_pad|>` placeholder to F slots, lookup text embeds via the HF model's `embed_tokens` weight, scatter audio embeds at the placeholder positions. 4. **CoreML prefill**: feed each token's embedding one-at-a-time to populate the in-model KV cache state. 5. **CoreML decode**: greedy step-by-step until `<|im_end|>`. The KV cache lives inside the CoreML model as `state`. Call `model.make_state()` once per request, then thread the same state object through every `predict()` call. ## Conversion details ```python # Apply per-op-name palettize: attention at LUT-8, MLP at LUT-4. prog = mlmodel._mil_program for op in prog.functions["main"].operations: if op.op_type != "const": continue n = op.name.lower() if "self_attn" in n and any(p in n for p in ("q_proj","k_proj","v_proj","o_proj")): attn_ops.append(op.name) elif "mlp" in n and any(p in n for p in ("gate_proj","up_proj","down_proj")): mlp_ops.append(op.name) config = OptimizationConfig(op_name_configs={ **{n: OpPalettizerConfig(nbits=8, group_size=8) for n in attn_ops}, **{n: OpPalettizerConfig(nbits=4, group_size=8) for n in mlp_ops}, }) mlmodel = palettize_weights(mlmodel, config) ``` The model exposes 84 attention weight ops (28 layers × 3 attention projections after the GQA-shared k/v gets clustered into k+v ops) and 84 MLP weight ops (28 layers × 3 MLP projections). `compute_precision=FLOAT32` is mandatory — fp16 compute on Qwen3-ASR produces all-NaN logits (RMSNorm + attention score overflow). A `coremltools` local patch was needed in `coremltools/converters/mil/frontend/torch/ops.py` `_cast`: numpy arrays of size 1 need to be coerced to scalar via `.flatten()[0].item()` before the dtype call — see `convert_embeds_mixed.py` setup notes. ## Known limitations 1. **ANE rejected**. CoreML's ANE compiler fails (`MILCompilerForANE error: failed to compile ANE model using ANEF`) — likely due to model size + stateful KV cache. `CPU_AND_NE` fails to load. `ALL` runs on **Metal GPU** (correct + ~3-4× faster than `CPU_ONLY`), which is the recommended setting. 2. **Audio encoder is ONNX**. The 24-layer Whisper-style encoder isn't ported to CoreML yet (ANEMLL is LLM-only). End-to-end runs the encoder via `onnxruntime` and the LLM via `coremltools`. 3. **Quality below ONNX/MLX** by ~2% at 4-bit, due to LUT k-means being weaker than GPTQ on this architecture. The uniform LUT-4 variant is smaller (826 MB) if size is critical; the mixed 8/4 (1.87 GB) is recommended for best quality. ## Companion repos - [Reza2kn/mega-asr-onnx](https://huggingface.co/Reza2kn/mega-asr-onnx) — full ONNX pipeline (GPTQ-INT4, 92.7%) - [Reza2kn/mega-asr-mlx](https://huggingface.co/Reza2kn/mega-asr-mlx) — MLX 4-bit (mixed 8/4 attn/MLP, 92.2%) - [Reza2kn/mega-asr-bench](https://huggingface.co/spaces/Reza2kn/mega-asr-bench) — browser demo (WebGPU) ## Credits - Original model: [zhifeixie/Mega-ASR](https://huggingface.co/zhifeixie/Mega-ASR) (1.7B, Apache-2.0) - CoreML conversion via [ANEMLL](https://github.com/Anemll/Anemll) with custom input_embeds + mixed-precision patches - Benchmark: [Voices-in-the-Wild-Bench](https://github.com/xzf-thu/Voices-in-the-Wild-Bench)