---
language:
- ro
library_name: pytorch
pipeline_tag: video-text-to-text
tags:
- visual-speech-recognition
- lip-reading
- vsr
- romanian
- speech-recognition
- audio-visual
datasets:
- vsro200/vsro200
metrics:
- wer
---

# VSRo-200: Romanian Visual Speech Recognition Models

This repository hosts the encoder-decoder VSR model checkpoints introduced in the paper *VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness*.

The models are MultiVSR backbones fine-tuned on the **VSRo-200** corpus, a 200-hour collection of Romanian podcast recordings. For training code, data preparation scripts, and inference instructions, please refer to the [GitHub repository](https://github.com/vsro200/vsro200).

## Checkpoints

All checkpoints follow the naming pattern `model_[hours]_[type].pt`:

- `_annot` — trained on human-annotated transcriptions
- `_auto` — trained on automatically generated pseudo-labels
- `_shuffle` — alternative data splits used for variance analysis (100h models)
- `_males` / `_females` / `_mix` — gender-controlled 40h subsets used for bias analysis

## Results

All results are reported in Word Error Rate (WER, %) and Character Error Rate (CER, %) on the **Test Unseen** and **Test Seen** splits. Lower is better.

#### Human Annotated Data

| Training Hours | Test Unseen WER (%) | Test Unseen CER (%) | Test Seen WER (%) | Test Seen CER (%) |
|:---:|:---:|:---:|:---:|:---:|
| 10h | 72.50 | 41.49 | 67.01 | 37.53 |
| 25h | 64.86 | 36.62 | 59.23 | 32.96 |
| 50h | 58.87 | 33.38 | 54.03 | 29.88 |
| 75h | 54.86 | 30.97 | 51.44 | 28.61 |
| 100h | **53.29** | **29.94** | **48.16** | **26.53** |

#### Whisper Pseudo Labels

| Training Hours | Test Unseen WER (%) | Test Unseen CER (%) | Test Seen WER (%) | Test Seen CER (%) |
|:---:|:---:|:---:|:---:|:---:|
| 10h | 74.61 | 42.09 | 68.41 | 38.22 |
| 25h | 66.27 | 37.05 | 60.40 | 33.36 |
| 50h | 59.28 | 33.15 | 55.39 | 30.65 |
| 75h | 56.25 | 31.18 | 51.56 | 28.33 |
| 100h | 53.63 | 30.12 | 49.61 | 27.22 |
| 125h | 51.71 | 29.04 | 48.68 | 26.58 |
| 150h | 51.25 | 28.40 | 47.05 | 25.64 |
| 175h | 49.84 | 27.66 | 46.44 | 25.30 |
| 200h | **48.75** | **27.05** | **44.54** | **24.51** |

A variance analysis across three random shuffles of the 100h subsets yields a mean Word Error Rate (WER) of 53.21% (± 0.37) for the human-annotated data and 53.82% (± 0.17) for the auto-generated data.



### Out-of-distribution robustness

*   **Test Seen / Unseen (In-Domain):** Baseline performance on podcast data, tested on our 200h-model.
*   **Vlogs:** Unconstrained videos shot in different camera angles, dynamic lighting, movement.
*   **Specific domains:** Content featuring highly specialized or technical vocabulary (e.g., medical, scientific). 
*   **Noisy:** Videos with poor resolution, bad lighting, or heavy motion blur.
*   **Archival (Black & White):** Historical footage with distinct visual artifacts, atypical framerates, and lack of color information.
*   **Global OOD:** The aggregated metrics across all out-of-distribution subsets.

| Dataset / Category | # Clips | WER (%) | CER (%) | OOV Token (%) | OOV Type (%) |
|:---|:---:|:---:|:---:|:---:|:---:|
| **Test Seen** | 386 | 44.54 | 24.51 | 1.67 | 6.93 |
| **Test Unseen** | 389 | 48.75 | 27.05 | 2.30 | 8.50 |
| **OOD: Vlogs** | 99 | 58.61 | 32.85 | 1.49 | 4.26 |
| **OOD: Specific domains** | 84 | 63.01 | 28.73 | 9.78 | 17.93 |
| **OOD: Noisy** | 100 | 68.96 | 33.68 | 6.19 | 12.88 |
| **OOD: Archival** | 92 | 87.97 | 50.44 | 5.24 | 10.96 |
| **Global OOD** | 375 | 68.46 | 35.99 | 5.08 | 14.75 |

#### Metrics Note
*   **Duration:** Each OOD category consists of 15 minutes of video content.
*   **OOV Token (%):** The percentage of *total words* in the evaluation set that do not appear in the training data. Measures how often unknown words occur.
*   **OOV Type (%):** The percentage of *unique words* in the evaluation set that do not appear in the training data. Measures the diversity of unknown words.

### Gender bias analysis (40h models)


To evaluate gender bias and cross-speaker generalization, we trained 40-hour baseline models on male-only, female-only, and mixed datasets. 

#### Test Unseen
| Training Set (40h) | Global WER (%) | Global CER (%) | Male WER (%) | Male CER (%) | Female WER (%) | Female CER (%) |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|
| Males Only | 62.15 | 35.23 | 61.32 | 34.51 | 62.97 | 35.95 |
| Females Only | **59.33** | **33.44** | **59.17** | **32.87** | **59.49** | **34.02** |
| Mixed Data | 59.52 | 33.74 | 59.19 | 33.26 | 59.85 | 34.22 |

#### Test Seen
| Training Set (40h) | Global WER (%) | Global CER (%) | Male WER (%) | Male CER (%) | Female WER (%) | Female CER (%) |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|
| Males Only | 58.82 | 33.11 | **58.58** | **32.59** | 59.06 | 33.63 |
| Females Only | 59.10 | 33.30 | 67.26 | 38.67 | **51.20** | **27.99** |
| Mixed Data | **56.29** | **31.22** | 60.56 | 33.54 | 52.15 | 28.93 |



## Citation

If you use these models, please cite:

```bibtex
@inproceedings{vsro200,
  title  = {VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness},
  author = {...},
  year   = {...}
}
```