🧏♂️ Sign Language Translation with Sentence Embedding Supervision
This repository contains resources related to the paper:
Sign Language Translation with Sentence Embedding Supervision
Yasser Hamidullah, Josef van Genabith, Cristina España-Bonet
Presented at the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), Short Papers
📄 Paper on ACL Anthology • DOI
📝 Abstract
State-of-the-art sign language translation (SLT) systems facilitate the learning process through gloss annotations, either in an end-to-end manner or by involving an intermediate step. Unfortunately, gloss-labelled sign language data is usually not available at scale and, when available, gloss annotations widely differ from dataset to dataset.
We present a novel approach using sentence embeddings of the target sentences at training time that take the role of glosses. This new form of supervision requires no manual annotation and can be learned from raw textual data.
Our method naturally supports multilingual settings, and we evaluate it on datasets covering German (PHOENIX-2014T) and American (How2Sign) sign languages. Experiments span both mono- and multilingual setups. Our approach significantly outperforms other gloss-free methods and narrows the performance gap with gloss-dependent systems, even without additional SLT pretraining data.
📊 Results Highlights
| Dataset | Approach | BLEURT ↑ |
|---|---|---|
| PHOENIX-2014T | Ours (no gloss) | 0.483 |
| How2Sign | Ours (no gloss) | 0.487 |
🚀 Coming Soon
We plan to release:
- Pre-trained models
- Inference notebooks or demo Spaces
📄 Citation
If you use this work, please cite:
@inproceedings{hamidullah-etal-2024-sign,
title = "Sign Language Translation with Sentence Embedding Supervision",
author = "Hamidullah, Yasser and
van Genabith, Josef and
Espa{\~n}a-Bonet, Cristina",
editor = "Ku, Lun-Wei and
Martins, Andre and
Srikumar, Vivek",
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.acl-short.40/",
doi = "10.18653/v1/2024.acl-short.40",
pages = "425--434"
}