| --- |
| license: apache-2.0 |
| language: |
| - ro |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - llama |
| - romanian |
| - synthetic-data |
| - distillation |
| - tinyfabulist |
| - fables |
| base_model: klusai/tf3-50m-base |
| datasets: |
| - klusai/ds-tf2-en-ro-15k |
| --- |
| |
| # TF3 Student: Distilled Romanian Language Model |
|
|
| A compact **22.9M-parameter** Romanian language model distilled from the [TF3-50M teacher](https://huggingface.co/klusai/tf3-50m-base) using logit-based knowledge distillation. Part of the [TinyFabulist](https://arxiv.org/abs/2601.10410) research project. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |----------|-------| |
| | Parameters | 22.9M (26.45M with untied embeddings) | |
| | Architecture | LLaMA-style decoder-only Transformer | |
| | Hidden size | 384 | |
| | Attention heads | 6 (head dim 64) | |
| | Layers | 6 | |
| | MLP intermediate | 1,024 | |
| | Vocab size | 32,000 (Unigram, Romanian-specific) | |
| | Context length | 2,048 tokens | |
| | Tied embeddings | Yes | |
| | Training | Knowledge distillation from klusai/tf3-50m-base | |
|
|
| ## Training |
|
|
| - **Method**: Logit-based knowledge distillation (KL + CE loss, alpha=0.009) |
| - **Teacher**: [klusai/tf3-50m-base](https://huggingface.co/klusai/tf3-50m-base) (51.65M params, frozen) |
| - **Data**: [klusai/ds-tf2-en-ro-15k](https://huggingface.co/datasets/klusai/ds-tf2-en-ro-15k) (15k Romanian fables) |
| - **Temperature**: T=1.0 |
| - **Epochs**: 3 |
| - **Learning rate**: 3e-4 (cosine schedule, 50-step warmup) |
| - **Hardware**: Apple M3 Ultra (96GB unified memory) |
|
|
| ## Intended Use |
|
|
| This model is a research artifact demonstrating knowledge distillation for compact Romanian language models trained on synthetic moral microfiction. It is designed for: |
|
|
| - Research on compact language model compression |
| - Romanian text generation in the fable/moral story domain |
| - Downstream fine-tuning for Romanian NLP tasks |
|
|
| **Not intended for**: Production text generation, factual question answering, or safety-critical applications. |
|
|
| ## Limitations |
|
|
| - Domain-restricted to moral microfiction (fables) |
| - Trained exclusively on synthetic data |
| - May exhibit repetitive patterns and simplified phrasing compared to the teacher |
| - Gender agreement errors may occur in generated text |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{nadas2026tf3, |
| title={TF3-RO-50M: Training Compact Romanian Language Models from Scratch on Synthetic Moral Microfiction}, |
| author={Nada\c{s}, Mihai Dan and Dio\c{s}an, Laura and Tomescu, Andreea and Pi\c{s}coran, Andrei}, |
| journal={arXiv preprint arXiv:2601.10410}, |
| year={2026} |
| } |
| ``` |
|
|
| ## Related Models and Datasets |
|
|
| | Artifact | Description | |
| |----------|-------------| |
| | [klusai/tf3-50m-base](https://huggingface.co/klusai/tf3-50m-base) | Teacher model (51.65M) | |
| | [klusai/tf3-50m-sft](https://huggingface.co/klusai/tf3-50m-sft) | SFT-tuned teacher | |
| | [klusai/tf3-bert](https://huggingface.co/klusai/tf3-bert) | NER model for entity coherence evaluation | |
| | [klusai/ds-tf2-en-ro-3m](https://huggingface.co/datasets/klusai/ds-tf2-en-ro-3m) | 3M bilingual fable corpus | |
| | [klusai/ds-tf2-en-ro-15k](https://huggingface.co/datasets/klusai/ds-tf2-en-ro-15k) | 15k curated subset for distillation/SFT | |
|
|
