GoldenNet-Qwen2.5-0.5B-LoRA-v1

Model Description

GoldenNet-Qwen2.5-0.5B-LoRA-v1 is a LoRA fine-tuned version of Qwen/Qwen2.5-0.5B-Instruct specialized for Iraqi Government Correspondence Processing.

This is the standard LoRA variant (no quantization), offering potentially better quality than QLoRA at the cost of higher VRAM usage during training.

Tasks

1. Document Classification - 8 categories (طلب، شكوى، تقرير، إعلام، استفسار، دعوة، تعميم، إحالة)
2. Named Entity Recognition - Extracts persons, organizations, locations, dates, monetary values, laws

Model Comparison

Model	Method	Train Loss	Eval Loss	Training Time
QLoRA-v1	4-bit QLoRA	0.448	0.2998	49s
LoRA-v1	Standard LoRA	0.496	0.3665	70s

Training Details

Parameter	Value
Base Model	Qwen/Qwen2.5-0.5B-Instruct
Fine-tuning Method	LoRA (no quantization)
LoRA Rank	64
LoRA Alpha	128
LoRA Dropout	0.05
Learning Rate	1e-4
Epochs	3
Batch Size	1 (effective: 16)
Max Sequence Length	2048
Precision	BF16
Trainable Parameters	35.2M (6.6%)

Loss Progression

• Epoch 1: 0.979
• Epoch 2: 0.350
• Epoch 3: 0.247

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Alamori/GoldenNet-Qwen2.5-0.5B-LoRA-v1",
    device_map="auto",
    torch_dtype="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Alamori/GoldenNet-Qwen2.5-0.5B-LoRA-v1")

# Classification example
correspondence = """جمهورية العراق
وزارة التربية
العدد: 1234/ت/2025

إلى/ السيد مدير عام التعليم المحترم

م/ طلب تعيين معلمين

نرجو الموافقة على تعيين 50 معلماً.

مع التقدير"""

instruction = "صنّف المراسلة الحكومية التالية إلى إحدى الفئات: طلب، شكوى، تقرير، إعلام، استفسار، دعوة، تعميم، إحالة. أجب بصيغة JSON."

messages = [{"role": "user", "content": f"{instruction}\n\n{correspondence}"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.1)
print(tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True))

Related Models

• GoldenNet-Qwen2.5-0.5B-QLoRA-v1 - 4-bit quantized version

License

Apache 2.0

---

Developed by Golden Net AI
Empowering Iraqi Government Digital Transformation