mrhuseyn4/qwen3-azerbaijani-lora

Qwen3-0.6B Fine-tuned for Azerbaijani

This model is a fine-tuned version of Qwen/Qwen3-0.6B on Azerbaijani instruction-following dataset.

Model Description

• Base Model: Qwen/Qwen3-0.6B
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Language: Azerbaijani (primary), English (preserved)
• Task: Instruction following and question answering

Training Details

• Training Steps: 10,000
• LoRA Rank: 16
• LoRA Alpha: 32
• Learning Rate: 1e-4
• Batch Size: 4 (per device) × 8 (gradient accumulation) = 32 (effective)
• Quantization: 4-bit (QLoRA)

Usage

Load with LoRA Adapter

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "mrhuseyn4/qwen3-azerbaijani-lora")
tokenizer = AutoTokenizer.from_pretrained("mrhuseyn4/qwen3-azerbaijani-lora", trust_remote_code=True)

# Generate
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Azərbaycanın paytaxtı hansı şəhərdir?"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.9
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Load Merged Model (if available)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "mrhuseyn4/qwen3-azerbaijani-lora",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("mrhuseyn4/qwen3-azerbaijani-lora", trust_remote_code=True)

# Use same generation code as above

Intended Use

This model is designed to:

• Answer questions in Azerbaijani
• Follow instructions in Azerbaijani
• Maintain general knowledge and reasoning capabilities from the base model

Limitations

• Primarily trained on Azerbaijani data
• May have reduced performance on other languages compared to base model
• Inherits limitations from the base Qwen3-0.6B model

Training Data

Fine-tuned on a large-scale Azerbaijani instruction-following dataset with ~800k examples.

Citation

If you use this model, please cite:

@misc{qwen3-azerbaijani,
  author = {Your Name},
  title = {Qwen3-0.6B Fine-tuned for Azerbaijani},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/mrhuseyn4/qwen3-azerbaijani-lora}}
}

Acknowledgments

• Base model: Qwen Team
• Fine-tuning framework: Hugging Face Transformers, PEFT