---
language:
- en
- kab
license: apache-2.0
library_name: sentence-transformers
tags:
- sentence-transformers
- kabyle
- taqbaylit
- tamazight
- berber
- embeddings
- cross-lingual
- african-languages
- nlp
datasets:
- Imsidag-community/nllb_en_kab
- Imsidag-community/english-kabyle-parallel
- Imsidag-community/libretranslate-suggestions
- ayymen/Weblate-Translations
pipeline_tag: sentence-similarity
---

# Kabyle Sentence Transformer (MPNet)

A sentence embedding model specifically fine-tuned for **Kabyle (Taqbaylit)** - **English** cross-lingual semantic similarity.

## Model Details

| Attribute | Value |
|-----------|-------|
| Base model | `sentence-transformers/paraphrase-multilingual-mpnet-base-v2` |
| Fine-tuning data | ~2.5M unique EN–KAB parallel sentences |
| Embedding dimension | 768 |
| Training framework | SentenceTransformers |
| Training time | ~1h 16min (1 epoch, 15,593 steps) |
| Final loss | 0.043 (started at 0.278) |

## Training Data

| Source | Pairs | Description |
|--------|-------|-------------|
| NLLB (cleaned) | ~2.35M | Diverse domain parallel corpus |
| Tatoeba + CS | ~202K | Community translations + software localization |
| Weblate | ~9K | FLOSS UI strings |
| LibreTranslate | ~449 | User-reviewed translations |

## Performance

Compared to the base `paraphrase-multilingual-mpnet-base-v2` (untrained):

| Metric | Base | This Model | Gain |
|--------|------|------------|------|
| Avg. cosine similarity (EN<->KAB) | 0.278 | **0.857** | **+58 points** |

## Usage

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("boffire/kabyle-sentence-transformer-mpnet")

# Embed English and Kabyle
sentences = ["Hello!", "Azul!"]
embeddings = model.encode(sentences)

# Cross-lingual similarity
from sklearn.metrics.pairwise import cosine_similarity
sim = cosine_similarity([embeddings[0]], [embeddings[1]])
print(sim)
```

## Limitations

- Trained primarily on parallel data; monolingual Kabyle similarity not explicitly optimized
- Best for EN<->KAB cross-lingual tasks; Kabyle<->Kabyle may work but is untested
- Religious text overrepresented in NLLB portion; may underperform on highly technical/modern domains
- Evaluator used constant labels (all 1.0) due to all pairs being positive; correlation metrics were undefined

## Future Work

- Train v2 with `Davlan/afro-xlmr-large` backbone for African-specific pretraining
- Add monolingual Kabyle data for better Kabyle<->Kabyle similarity
- Fix evaluator to use `AvgCosineEvaluator` instead of correlation-based metrics
- Evaluate against LASER on a proper benchmark

## Citation

If you use this model, please cite:

```bibtex
@misc{kabyle-st-mpnet,
  title={Kabyle Sentence Transformer},
  author={boffire},
  year={2026},
  howpublished={\url{https://huggingface.co/boffire/kabyle-sentence-transformer-mpnet}}
}
```

## Acknowledgments

- Imsidag-community for the cleaned parallel corpora
- Tatoeba contributors for community translations
- Meta AI for LASER and NLLB datasets
- boffire community for Kabyle NLP tooling