moushi21/dpo-qwen-cot-merged20

Qwen3-4B-StructEval-Iterative-Alignment

This model is a highly optimized version of Qwen3-4B-Instruct-2507, specifically developed for the StructEval-T task. It features a sophisticated iterative alignment process to maximize performance in structured data reasoning and Chain-of-Thought (CoT) generation.

This repository contains full-merged 16-bit weights. No adapter loading is required.

Training Pipeline

Unlike standard fine-tuning, this model has undergone a four-stage iterative training process to ensure precise alignment and deep reasoning capabilities:

1. Stage 1: SFT - Foundation building with structured CoT trajectories.
2. Stage 2: DPO - First alignment to preference reasoning paths.
3. Stage 3: SFT - Knowledge reinforcement and format refinement.
4. Stage 4: DPO - Final preference optimization for high-fidelity structured outputs.

Training Objective

The model is engineered to excel in:

• Complex Reasoning: Enhanced Chain-of-Thought processing for structural evaluation.
• Structural Integrity: Strict adherence to complex data formats (JSON, Tables, etc.).
• Consistency: High-reliability outputs across iterative multi-turn interactions.

Training Configuration (Final Stage)

• Method: Iterative DPO (Direct Preference Optimization)
• Base model: unsloth/Qwen3-4B-Instruct-2507
• Epochs: 1
• Learning rate: 3e-06
• Beta: 0.05
• Max sequence length: 2560
• Platform: Trained with Unsloth

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "moushi21/dpo-qwen-cot-merged20"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

Sources & Terms (IMPORTANT)

Training data:

• u-10bei/structured_data_with_cot_dataset_512_v2
• u-10bei/structured_data_with_cot_dataset_512_v4
• u-10bei/structured_data_with_cot_dataset_512_v5
• u-10bei/structured_data_with_cot_dataset_512
• u-10bei/structured_data_with_cot_dataset_v2
• u-10bei/structured_data_with_cot_dataset
• u-10bei/dpo-dataset-qwen-cot

Dataset License: MIT License. This dataset is used and distributed under the terms of the MIT License. Compliance: Users must comply with the MIT license (including copyright notice) and the base model's original terms of use.