Hzzzzx0/qwen3-0.6b-oral-lora

Qwen3-0.6B Oral Translation LoRA

This is a LoRA adapter for Qwen/Qwen3-0.6B fine-tuned for bilingual oral translation (Chinese ↔ English).

Model Details

• Base Model: Qwen/Qwen3-0.6B
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• LoRA Config: rank=16, alpha=32, dropout=0.05
• Target Modules: q_proj, k_proj, v_proj, o_proj
• Training Data: OpenSubtitles en-zh (5,000 samples)
• Training Time: ~6 minutes (dual GPU)

Performance

Model	BLEU Score	Improvement
Baseline (Qwen3-0.6B)	1.24	-
LoRA Fine-tuned	11.89	+858%

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "Hzzzzx0/qwen3-0.6b-oral-lora")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B", trust_remote_code=True)

# Translate
def translate(text, direction="zh2en"):
    if direction == "zh2en":
        inst = "请把下面中文翻译成口语自然的英文。只输出译文。"
    else:
        inst = "请把下面英文翻译成口语自然的中文。只输出译文。"
    
    prompt = f"### Instruction:\n{inst}\n\n### Input:\n{text}\n\n### Response:\n"
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=128,
            do_sample=False,
            repetition_penalty=1.2,
            eos_token_id=tokenizer.eos_token_id,
            pad_token_id=tokenizer.pad_token_id,
        )
    
    result = tokenizer.decode(output[0], skip_special_tokens=True)
    return result.split("### Response:")[-1].strip()

# Example
print(translate("你好呀", "zh2en"))  # Output: Hi, how are you?

Training Details

• Epochs: 3
• Batch Size: 24 per device
• Gradient Accumulation: 3 steps
• Effective Batch Size: 72
• Learning Rate: 2e-4
• Warmup Ratio: 0.03
• Precision: bfloat16 (full precision, no quantization)

Example Translations

Chinese → English:

• 输入: 你好呀 → Output: Hi, how are you?
• 输入: 今天天气真不错 → Output: The weather is really nice today

English → Chinese:

• 输入: See you later → Output: 回头见。
• 输入: How are you doing? → Output: 你还好吗？

Limitations

• Trained on subtitle data, may not perform well on formal/technical text
• Limited to conversational/oral style translation
• Small training dataset (5K samples)
• May struggle with domain-specific terminology

Citation

@misc{qwen3-oral-lora,
  title={Qwen3-0.6B Oral Translation LoRA},
  author={Hzzzzx0},
  year={2025},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/Hzzzzx0/qwen3-0.6b-oral-lora}}
}

License

Apache 2.0

Links

• Demo Space
• GitHub Repository