blackhao0426/pref-extractor-qwen3-0.6b-full-sft

Preference Extractor (Qwen3-0.6B Full SFT)

A lightweight preference extraction model finetuned from Qwen3-0.6B. Given a dialogue window, it outputs structured JSON preference tuples describing user style preferences.

This model was introduced in the paper User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction. It is a core component of VARS (Vector-Adapted Retrieval Scoring), a framework for personalized LLM assistants.

Task

Given recent conversation turns between a user and an AI assistant, extract user preferences as structured condition-action rules in JSON format:

{
  "preferences": [
    {"condition": "when explaining math", "action": "show step-by-step derivation", "confidence": 0.9},
    {"condition": "general", "action": "respond in Chinese", "confidence": 0.85}
  ]
}

Each preference is a (condition, action, confidence) tuple:

• condition: when this preference applies (e.g., "when writing code", "general", "when debugging")
• action: the preferred behavior (e.g., "use Python with type hints", "show step-by-step derivation")
• confidence: extraction confidence score (0.0–1.0)

Performance

Evaluated on a held-out test set:

Metric	Value
JSON validity	99.7%
Recall	97.5%
Precision	37.7%
Eval loss	0.1611

The model intentionally favors high recall over precision: it over-extracts candidate preferences, and the downstream reranker and user vector in the VARS pipeline filter irrelevant ones.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "blackhao0426/pref-extractor-qwen3-0.6b-full-sft"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="bfloat16", device_map="auto", trust_remote_code=True)

# system_prompt should describe the extraction task
system_prompt = "You are a preference extractor. Given the dialogue window, extract preferences into JSON format."

conversation = """User: Can you explain the quicksort algorithm? I prefer step-by-step breakdowns with Python code.
Assistant: Sure! Here's quicksort step by step...
User: Great, but can you add type hints to the code?"""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": conversation},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)

# Decode only the newly generated tokens
result = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
# Parse JSON from result to get preference list

Training

Parameter	Value
Base model	Qwen/Qwen3-0.6B
Training data	blackhao0426/user-preference-564k (564K examples)
Learning rate	2e-05
Batch size	128 (32 per device x 4 GPUs)
Epochs	1
LR scheduler	Cosine with 5% warmup
Optimizer	AdamW (fused)
Framework	LLaMA-Factory

Citation

@misc{hao2026userpreferencemodelingconversational,
      title={User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction},
      author={Yuren Hao and Shuhaib Mehri and ChengXiang Zhai and Dilek Hakkani-Tür},
      year={2026},
      eprint={2603.20939},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.20939},
}