README.md

---
base_model: tocico28/MNLP_M3_dpo_model
datasets: youssefbelghmi/MNLP_M3_mcqa_dataset
library_name: transformers
model_name: MNLP_M3_dpo_mcqa_model
tags:
- generated_from_trainer
- trl
- sft
licence: license
---

# MNLP M3 MCQA Model

This model is a fine-tuned version of [tocico28/MNLP_M3_dpo_model](https://huggingface.co/tocico28/MNLP_M3_dpo_model) on the [youssefbelghmi/MNLP_M3_mcqa_dataset](https://huggingface.co/datasets/youssefbelghmi/MNLP_M3_mcqa_dataset), a large-scale collection of multiple-choice questions designed for evaluating and training models in **STEM** domains (science, math, engineering, medicine, etc.).

The [tocico28/MNLP_M3_dpo_model](https://huggingface.co/tocico28/MNLP_M3_dpo_model) is itself a fine-tuned version of **Qwen/Qwen3-0.6B-Base** using a dataset of preference-labeled STEM response pairs collected through a collaborative classroom annotation effort.

It has been trained using [TRL](https://github.com/huggingface/trl) as part of the final milestone of the **CS-552: Modern NLP** course at EPFL (Spring 2025).

## Task

**Multiple-Choice Question Answering (MCQA):** Given a question and four answer options (A–D), the model must complete the prompt with the correct option letter only (e.g., `A`, `B`, `C`, or `D`). It was trained with rationales during supervision but outputs only the letter during inference, making it compatible with evaluation frameworks such as LightEval.

## Training Dataset

- **Dataset:** [`youssefbelghmi/MNLP_M3_mcqa_dataset`](https://huggingface.co/datasets/youssefbelghmi/MNLP_M3_mcqa_dataset).
- ~30,000 questions from SciQ, OpenBookQA, MathQA, ARC, and MedMCQA.
- Each sample includes in particular:
  - question,
  - four answer choices (A–D),
  - the correct answer as a letter,
  - a short explanation (`support`) to guide learning.

## Training Setup

- **Base model:** `Qwen/Qwen3-0.6B-Base`.
- **Method:** Supervised Fine-Tuning (SFT) with `trl` and `SFTTrainer`.
- **Tokenizer:** AutoTokenizer (with `eos_token` used as padding).

## Training Prompt Format

During fine-tuning, each training example is converted into a prompt-completion pair. The prompt includes both the question and an explanation to guide the model’s reasoning:

```text
The following is a multiple-choice question (with answers) about knowledge and skills in advanced master's-level STEM fields.
You will be provided with an explanation to help you understand the correct answer.
Select the correct answer by replying with the option letter (A, B, C, or D) only.
Question: <question_text>
A. <option_A>
B. <option_B>
C. <option_C>
D. <option_D>
Explanation: <support_text>
Answer:
```

The completion is a single token: " A", " B", " C", or " D", corresponding to the correct answer.

## Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-5
- num_train_epochs: 1
- per_device_train_batch_size: 4
- per_device_eval_batch_size: 4
- gradient_accumulation_steps: 4
- gradient_checkpointing: true
- eval_strategy: steps
- eval_steps: 100
- logging_steps: 100
## Training Results
|   Epoch |   Training Loss |   Validation Loss |
|--------:|----------------:|------------------:|
|    0.08 |          0.3363 |            0.2766 |
|    0.15 |          0.2938 |            0.2719 |
|    0.23 |          0.2817 |            0.2751 |
|    0.31 |          0.2688 |            0.2604 |
|    0.38 |          0.2692 |            0.2640 |
|    0.46 |          0.2611 |            0.2571 |
|    0.54 |          0.2431 |            0.2433 |
|    0.61 |          0.2495 |            0.2439 |
|    0.69 |          0.2489 |            0.2384 |
|    0.77 |          0.2321 |            0.2376 |
|    0.84 |          0.2363 |            0.2353 |
|    0.92 |          0.2106 |            0.2358 |
|    0.99 |          0.2091 |            0.2340 | 
- **Final validation accuracy:** ~92.0%  
### Framework versions
- TRL: 0.17.0
- Transformers: 4.53.0.dev0
- Pytorch: 2.7.0
- Datasets: 3.2.0
- Tokenizers: 0.21.0
## Citations
Cite TRL as:
    
```bibtex
@misc{vonwerra2022trl,
	title        = {{TRL: Transformer Reinforcement Learning}},
	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
	year         = 2020,
	journal      = {GitHub repository},
	publisher    = {GitHub},
	howpublished = {\url{https://github.com/huggingface/trl}}
}
```
## Author
Developed by [**Youssef Belghmi**](https://huggingface.co/youssefbelghmi)  
CS-552: Modern NLP – EPFL, Spring 2025