---
language:
- kab
tags:
- gpt2
- causal-lm
- custom-vocab
license: mit
datasets:
- custom-kabyle-corpus
metrics:
- perplexity
---

# Kabyle GPT-2 Base Model (Optimized BPE)

This is a custom, lightweight GPT-2 style causal language model built from scratch specifically for the **Kabyle (Taqbaylit)** language. It utilizes a highly optimized morphological subword tokenizer trained with byte-aware rules to natively preserve and parse Latin-Tamazight text structures without visual noise artifacts.

## Model Highlights
* **Architecture:** Custom 8-layer, 8-attention-head Transformer (512 hidden dimensions) built from scratch.
* **Context Window:** 256 tokens.
* **Vocabulary Size:** 50,257 tokens.
* **Tokenizer Efficiency:** Achieves an exceptional **97.95% vocabulary utilization rate** on native Kabyle corpuses, maximizing embedding row saturation and eliminating dead parameters common in massive multilingual tokenizers.

## Tokenizer Performance

Our custom Byte-Pair Encoding (BPE) pipeline maps linguistic affixes accurately. Compared to standard tokenizers that introduce raw byte visual noise (e.g., `ÉĽ`, `áºĵ`), this model correctly keeps character boundaries intact during inference:

| Input Text Fragment | Standard Decoders (Noisy) | Our Native Pipeline (Clean) |
| :--- | :--- | :--- |
| **... yettɛawad ...** | `['yett', 'ÉĽawad']` | `['yett', 'ɛawad']` |
| **... iẓerfan ...** | `['Ġiáºĵer', 'fan']` | `['Ġiẓer', 'fan']` |

## Quickstart Usage

You can load this model and its accompanying optimized tokenizer directly into your PyTorch environment:

```python
from transformers import PreTrainedTokenizerFast, GPT2LMHeadModel

# Load the custom assets
tokenizer = PreTrainedTokenizerFast.from_pretrained("boffire/kabyle-gpt2-tokenizer")
model = GPT2LMHeadModel.from_pretrained("your-username/kabyle-llm-base")

# Quick inference test
text = "Wa d amcic-is aberkan,"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=40, do_sample=True, top_k=50)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

## Training Data & Methodology

The model was pre-trained using a meticulously cleaned and normalized **Kabyle text corpus** (~20 MB / 5.01M total tokens). 

### Optimization Settings
* **Training Duration:** 3 Epochs
* **Optimizer:** AdamW
* **Learning Rate:** `5e-4`
* **Batch Strategy:** Dynamic batch padding to maximize hardware and VRAM efficiency.