Lekhansh/qwen-3-8b-embedding-miti-global

Qwen 3 8B Embedding - MI Fidelity Global Scorer

This model is a fine-tuned version of Qwen/Qwen3-Embedding-8B for multi-output regression on Motivational Interviewing (MI) fidelity scoring.

Model Description

This model performs multi-output regression to score MI counseling transcripts on four global dimensions:

1. Cultivating Change Talk: How well the counselor elicits and reinforces change talk
2. Softening Sustain Talk: How well the counselor responds to sustain talk
3. Partnership: The collaborative nature of the interaction
4. Empathy: The counselor's understanding and empathy

Each dimension is scored on a scale of 1-5.

Training Details

Training Data

• Custom MI transcript dataset with expert annotations
• Each transcript is annotated with scores for all 4 dimensions

Training Procedure

• Base Model: Qwen/Qwen3-Embedding-8B
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Training Objective: Multi-output regression (4 continuous outputs)
• Sequence Length: 9000 tokens
• Quantization: 4-bit quantization during training
• LoRA Configuration:
  • r: 32
  • alpha: 64
  • dropout: 0.05
  • target_modules: q_proj, k_proj, v_proj, o_proj, up_proj, down_proj

Model Architecture

• The model uses a sequence classification head with 4 outputs (one per dimension)
• Input: Task instruction + Annotator ID + Transcript
• Output: 4-dimensional vector of scores [cultivating_change_talk, softening_sustain_talk, partnership, empathy]

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import json

# Load model and tokenizer
model_name = "Lekhansh/qwen-3-8b-embedding-miti-global"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Prepare input
dimension_names = ["cultivating change talk", "softening sustain talk", "partnership", "empathy"]
task_instruction = (
    f"Task: Score this transcript on the following dimensions from 1 to 5: "
    f"{', '.join(dimension_names)}. "
    f"Return scores in order: [{', '.join(dimension_names)}]"
)

transcript = "YOUR_MI_TRANSCRIPT_HERE"
annotator_id = "annotator_001"  # Optional: provide annotator identity

input_text = task_instruction + "\n" + f"Annotator: {{annotator_id}}" + "\n" + transcript

# Tokenize
inputs = tokenizer(
    input_text,
    padding='max_length',
    truncation=True,
    max_length=9000,
    return_tensors="pt"
).to(model.device)

# Get predictions
with torch.no_grad():
    outputs = model(**inputs)
    predictions = outputs.logits[0].cpu().numpy()

# Parse predictions
scores = {{
    "cultivating_change_talk": float(predictions[0]),
    "softening_sustain_talk": float(predictions[1]),
    "partnership": float(predictions[2]),
    "empathy": float(predictions[3])
}}

print(json.dumps(scores, indent=2))

Output Format

The model outputs a 4-dimensional vector where each value is a continuous score between 1 and 5:

• Index 0: Cultivating Change Talk
• Index 1: Softening Sustain Talk
• Index 2: Partnership
• Index 3: Empathy

Evaluation Metrics

The model is evaluated using:

• MAE (Mean Absolute Error): Lower is better
• RMSE (Root Mean Squared Error): Lower is better
• Pearson R: Correlation between predictions and ground truth
• Within-1 Accuracy: Percentage of predictions within 1.0 of the true value
• Exact Accuracy (Rounded): Percentage of predictions that match after rounding

Metrics are computed both globally (across all dimensions) and per-dimension.

Limitations and Bias

• This model is trained on specific MI transcript data and may not generalize to other counseling modalities
• The model's performance depends on the quality and diversity of the training annotations
• Annotator identity is included in the input, which may introduce annotator-specific biases

Citation

If you use this model, please cite:

@misc{qwen3-mi-fidelity-scorer,
  author = {Lekhansh Shukla},
  title = {Qwen 3 8B Embedding - MI Fidelity Global Scorer},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Lekhansh/qwen-3-8b-embedding-miti-global}
}

License

This model is released under the Apache 2.0 license, following the base model's licensing.