qwen3-4b-struct-lora-v4-merged

This repository provides a LoRA-merged base model derived from:

Qwen/Qwen3-4B-Instruct-2507

This model is created by merging a structured-output LoRA adapter into the base model and serves as a foundation model for further fine-tuning (v11).

This is NOT a LoRA adapter.
This is a fully merged, standalone model.

---

What is this model?

This model is a structured-output specialized base model.

It was created by:

1. Fine-tuning Qwen3-4B-Instruct with LoRA for structured outputs
2. Merging the LoRA weights into the base model
3. Publishing the merged result as a new foundation checkpoint

This allows future LoRA training to start from a better structured-output prior.

Why this merged model exists

Typical workflow:

Qwen3-4B-Instruct

↓ LoRA (structured output training)

↓ merge

qwen3-4b-struct-lora-v4-merged ← this repo

↓ new LoRA training (v11)

This model significantly stabilizes later SFT and improves convergence.

---

Training Configuration

Item	Value
Base model	Qwen/Qwen3-4B-Instruct-2507
Method	LoRA SFT (no quantization, bf16)
Max sequence length	4096
Epochs	1
Learning rate	3e-5
Warmup ratio	0.1
Weight decay	0.05
LoRA r	64
LoRA alpha	128
LoRA dropout	0.05
Target modules	q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
Mask CoT	Yes (after_marker)
Dataset	u-10bei/structured_data_with_cot_dataset_512_v2

---

Usage

This is a fully merged standalone model. No LoRA adapter is required.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "azuki-digital/qwen3-4b-struct-lora-v4-merged"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

---

License & Compliance

This model inherits the license of:

• Qwen3-4B-Instruct (base model)
• Structured-output dataset used during original LoRA training

Users must comply with the original base model terms.