| --- |
| license: mit |
| datasets: |
| - westbrook/LibriMix |
| language: |
| - en |
| tags: |
| - speech |
| - SE |
| - Neural-Audio-Codec |
| pipeline_tag: audio-to-audio |
| --- |
| # Modeling strategies for speech enhancement in the latent space of a neural audio codec |
|
|
| This repository provides the official model checkpoints for the paper *[Modeling strategies for speech enhancement in the latent space of a neural audio codec](https://arxiv.org/abs/2510.26299)* authored by Sofiene Kammoun, Xavier Alameda-Pineda, and Simon Leglaive, and published at IEEE ICASSP 2026. |
|
|
| We explore different modeling strategies (autoregressive vs. non-autoregressive) and representation spaces (discrete vs. continuous) for speech enhancement using neural audio codecs and Conformer-based architectures. |
|
|
| [arXiv](https://arxiv.org/abs/2510.26299) | [Code and Audio examples](https://sofienekammoun.github.io/SE-NAC-25/) | [Bibtex](#citation) |
|
|
|
|
| ## Overview |
|
|
| Our work introduces and compares a family of speech enhancement models that systematically vary along two main axes: |
|
|
| - **Representation Type** |
| - Discrete tokens |
| - Continuous latent vectors |
|
|
| - **Modeling Strategy** |
| - Autoregressive (AR): Sequential prediction of clean speech representation |
| - Non-Autoregressive (NAR): Parallel prediction of clean speech representation |
|
|
| The current release includes the following models: |
|
|
| | Model Name | Modeling Strategy | Input Representation | Output Representation | Model Checkpoint | |
| |-------------|------|----------------|----------------|----------------| |
| | **D-AR** | Autoregressive | Discrete |Discrete | `D-AR_ckpt_300.pt` | |
| | **D-NAR** | Non-Autoregressive | Discrete |Discrete | `D-NAR_ckpt_300.pt` | |
| | **D-NAR*** | Non-Autoregressive | Continuous |Discrete | `D-NAR_star_ckpt_300.pt` | |
| | **C-AR** | Autoregressive | Continuous | Continuous | `C-AR_ckpt_300.pt` | |
| | **C-NAR** | Non-Autoregressive | Continuous | Continuous | `C-NAR_ckpt_300.pt` | |
|
|
| Additional models: |
| - **C-FT** (`C-FT-encoder_ckpt_300.pt`) and **D-FT** (`D-FT-encoder_ckpt_300.pt`), where we only finetune the NAC's encoder with an MSE loss and a cross-entropy loss, respectively. |
| - **STFT-NAR** (`STFT_NAR_Mask_ckpt_300.pt`), where instead of the embeddings of the NAC, we work with STFT representations, and we train the model to output an STFT mask. |
