Nishef/SmolLM2-360M-Full_ORPO_20251225_062447-merged

SmolLM2-360M - Orpo

Model Description

This model is a Merged Standalone Model fine-tuned from HuggingFaceTB/SmolLM2-360M using the Orpo training method.

Odds Ratio Preference Optimization - Reference-free optimization using odds ratios

This model was developed as part of thesis research on LLM Alignment using Preference Optimization Methods.

Model Details

Property	Value
Base Model	HuggingFaceTB/SmolLM2-360M
Training Method	Orpo
Model Type	Merged Standalone Model
Training Date	December 2025
Framework	PyTorch + Transformers + PEFT

Benchmark Results

Benchmark	Score
HellaSwag (10-shot)	0.526
TruthfulQA (0-shot MC2)	0.373
MMLU-Mini (5-shot)	0.249

Comparative Analysis

The following chart compares this method against other training approaches on the same base model:

Training Configuration

Parameter	Value
Epochs	1
Batch Size	2
Gradient Accumulation	8
Effective Batch Size	16
Learning Rate	2e-4
Max Sequence Length	512
LoRA Rank	16
LoRA Alpha	32
Dataset	Combined Preference Dataset (HH-RLHF + SHP + OpenAssistant)

Combined Preference Dataset (kto_combined)

Training uses a Combined Preference Dataset built via Round-Robin Sampling from three sources:

Source	Total Samples	Interactions
Anthropic HH-RLHF	321,600	61,568
Stanford Human Preferences (SHP)	697,436	38,984
OpenAssistant Conversations v1	16,810	8,904
Total	1,035,846	109,456

Actual Training Statistics (subset split train_prefs[:32090]):

• Training samples: 13,300 (paired examples)
• Validation samples: 700 (5%)
• Round-Robin distribution: 1,130 interactions per source
• Seed: 42 (for reproducibility)

Usage

Direct Loading (Merged Model)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("SmolLM2-360M")
tokenizer = AutoTokenizer.from_pretrained("SmolLM2-360M")

# Generate text
inputs = tokenizer("Hello, how are you?", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))

Training Methodology

Orpo

Odds Ratio Preference Optimization - Reference-free optimization using odds ratios

Key Features:

• Reference-free optimization
• Odds ratio-based preference modeling
• Reduced memory requirements (no reference model)
• Efficient training dynamics

Citation

If you use this model in your research, please cite:

@misc{smollm2_360m_orpo_2025,
  title = {SmolLM2-360M Fine-tuned with Orpo},
  author = {Thesis Research},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/Nishef/SmolLM2-360M-Full_ORPO_20251225_062447}
}

Repository Structure

.
├── adapter_config.json      # LoRA configuration
├── adapter_model.safetensors # Model weights
├── tokenizer files          # Tokenizer configuration
├── eval_summary.csv         # Evaluation results
├── thesis_plots/            # Visualization assets
│   ├── benchmark_results.png
│   └── training_loss.png
└── README.md               # This file

Acknowledgments

• Base Model: HuggingFaceTB/SmolLM2-360M
• Training Framework: Hugging Face Transformers
• Fine-tuning Library: PEFT

License

This model is released under the Apache 2.0 license.

---

This model was created as part of thesis research on LLM alignment using preference optimization methods.