Qwen-TS-500M

A 500M parameter language model specialised in 3GPP and ETSI telecommunications standards, trained via full fine-tuning on TeleSpec-Data.

Part of the tele-SLMs series — small language models adapted exclusively to telecommunications standards documents, with zero arXiv or web content in the training corpus.

Instruction-tuned version coming soon: Qwen-TS-500M-it

---

Model Details

Base model	Qwen/Qwen2.5-0.5B
Parameters	494M
Training	Full fine-tuning on TeleSpec-Data
Pretraining data	TeleSpec-Data (1.87B tokens)
Context length	4096 tokens
Hardware	2× NVIDIA RTX 6000 Ada Generation (48GB) + DeepSpeed ZeRO-2

---

Training

Full fine-tuning of all model weights on 409,117 packed 4096-token blocks (1.67B tokens) from 38,302 standards documents — 15,054 3GPP (Rel-8 to Rel-19) and 23,248 ETSI documents spanning 15 working groups (2000–2024). Zero arXiv or web content — 100% standards text.

• Epochs: 2
• Effective batch size: 128 — LR: 5e-5 (cosine with warmup)
• Context length: 4096 tokens
• DeepSpeed ZeRO-2 for memory efficiency

---

Usage

This is a base model — it continues text rather than following instructions. An instruction-tuned version will be released shortly.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id  = "nareshmodina/Qwen-TS-500M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model     = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

prompt  = "The RRC Connection Establishment procedure in LTE is"
inputs  = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Limitations

• Base model only — does not follow instructions
• Standards only — strong 3GPP/ETSI knowledge, limited general telecom knowledge
• Not for production — intended for research purposes only

---

Links

• 📦 Dataset: nareshmodina/TeleSpec-Data
• 📊 Benchmark: AliMaatouk/Tele-Eval
• 🗂️ Collection: nareshmodina/SmolLM-TS

---

Citation

@misc{modina2025teleslms,
  author    = {Naresh Modina},
  title     = {tele-SLMs: Small Language Models for Telecommunications Standards},
  year      = {2025},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/nareshmodina/Qwen-TS-500M}
}