Llama 3.2 3B — Medical Q&A

Fine-tuned version of Meta's Llama 3.2 3B Instruct on the PubMedQA biomedical dataset.

Model Description

This model is specialized for biomedical question answering. Given a research context and a medical question, it answers yes, no, or maybe with a detailed explanation.

Training Details

Parameter	Value
Base model	Llama 3.2 3B Instruct
Dataset	PubMedQA (pqa_labeled)
Training samples	1,000
Technique	QLoRA (4-bit quantization + LoRA)
LoRA rank	16
Epochs	3
Learning rate	2e-4
Final training loss	1.56
Training time	~7 minutes (A100)
Trainable parameters	~13M / 3B (0.42%)

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Yashwanth-Pulimi/llama3-medical-qa"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)

prompt = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are an expert medical AI assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
Context: A randomized trial of 200 diabetic patients showed metformin 
reduced HbA1c by 2.1% vs 1.2% with lifestyle changes alone.

Question: Does metformin reduce HbA1c more than lifestyle changes alone?<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>"""

inputs  = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=150, temperature=0.1, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Dataset

PubMedQA is a biomedical question answering dataset collected from PubMed abstracts. Questions are answered with yes, no, or maybe based on research context.