Qwen-TS-500M-it

A 500M parameter instruction-tuned language model specialised in 3GPP and ETSI telecommunications standards. Trained via full fine-tuning on TeleSpec-Data followed by LoRA instruction fine-tuning on Alpaca.

Part of the tele-SLMs series — small language models adapted exclusively to telecommunications standards documents, with zero arXiv or web content in the training corpus.

Looking for the base pretrained version? See nareshmodina/Qwen-TS-500M

---

Model Details

Base model	Qwen/Qwen2.5-0.5B
Parameters	494M
Training	Full FT pretrain → LoRA SFT (Alpaca)
Pretraining data	TeleSpec-Data (1.87B tokens)
SFT data	Alpaca 52k (full dataset)
Context length	4096 tokens
Hardware	2× NVIDIA RTX 6000 Ada Generation (48GB) + DeepSpeed ZeRO-2

---

Training

Stage 1 — Full fine-tuning on TeleSpec-Data

All model weights updated on 409,117 packed 4096-token blocks (1.67B tokens) from 38,302 standards documents — 15,054 3GPP (Rel-8 to Rel-19) and 23,248 ETSI documents spanning 15 working groups (2000–2024). Zero arXiv or web content — 100% standards text.

• Epochs: 2 — Effective batch size: 128 — LR: 5e-5 (cosine)
• DeepSpeed ZeRO-2 for memory efficiency on 2×48GB GPUs

Stage 2 — LoRA instruction fine-tuning

LoRA (r=16, α=32) on full Alpaca 52k dataset. Base weights frozen to preserve domain knowledge.

• Epochs: 1 — LR: 1e-5

---

Evaluation

Evaluated on Tele-Eval using the metrics defined in Maatouk et al. (2024) — standards-derived questions only (standard_* IDs, 10,000 examples, seed 42).

Model	Ans-PPL ↓	SemScore ↑
Qwen2.5-0.5B-alpaca (base + Alpaca SFT)	10.57	0.6123
Qwen-TS-500M-it (ours)	5.39	0.6901

49.0% Ans-PPL reduction vs base+SFT baseline — strongest improvement in the series.

Comparison across model sizes:

Model	Ans-PPL ↓	SemScore ↑
SmolLM-TS-135M-it	9.19	0.6504
SmolLM-TS-360M-it	8.62	0.6572
Qwen-TS-500M-it	5.39	0.6901

---

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id  = "nareshmodina/Qwen-TS-500M-it"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model     = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

prompt = (
    "The following is a question about telecommunications and networking.\n"
    "Question: What is the purpose of the RRC Connection Establishment procedure in LTE?\n"
    "Answer:"
)

inputs  = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=False,
    repetition_penalty=1.3,
)
answer = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(answer)

Note: Use the Alpaca-style Question: ... Answer: prompt format for best results.

---

Limitations

• Alpaca SFT — trained for Q&A style responses, not multi-turn conversation
• Standards only — strong 3GPP/ETSI knowledge, limited general telecom knowledge
• Not for production — intended for research purposes only

---

Links

• 📦 Dataset: nareshmodina/TeleSpec-Data
• 🤖 Base model: nareshmodina/Qwen-TS-500M
• 📊 Benchmark: AliMaatouk/Tele-Eval
• 🗂️ Collection: nareshmodina/SmolLM-TS

---

Citation

@misc{modina2025teleslms,
  author    = {Naresh Modina},
  title     = {tele-SLMs: Small Language Models for Telecommunications Standards},
  year      = {2025},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/nareshmodina/Qwen-TS-500M-it}
}

@misc{maatouk2024telellms,
  title         = {Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications},
  author        = {Ali Maatouk and Kenny Chirino Ampudia and Rex Ying and Leandros Tassiulas},
  year          = {2024},
  eprint        = {2409.05314},
  archivePrefix = {arXiv},
  primaryClass  = {cs.IT}
}