twkang43/lora-roberta-cse4057

Model Card for Model ID

LoRA-based fine-tuned RoBERTa model for multi-intent classification in natural language utterances.

---

Model Details

Model Description

• Developed by: Taewook Kang (Github: @twkang43)
• Model type: Fine-tuned RoBERTa model with Low-Rank Adaptation (LoRA) for multiple intent classification.
• Language(s) (NLP): English
• License: MIT License
• Finetuned from model: RoBERTa-base

Uses

How to Get Started with the Model

Use the code below to download the tokenizer and fine-tuned model with LoRA.

from transformers import AutoTokenizer,AutoModelForSequenceClassification
from peft import PeftModel

tokenizer = AutoTokenizer.from_pretrained("twkang43/lora-roberta-cse4057")
base_model = AutoModelForSequenceClassification.from_pretrained(
    MODEL_NAME,
    problem_type="multi_label_classification",
    num_labels=num_labels,
    id2label=id2label,
    label2id=label2id
).to(DEVICE)
model = PeftModel.from_pretrained(base_model, "twkang43/lora-roberta-cse4057")

Training Details

Training Data

Fine-tuned on the BlendX dataset ("train" dataset was split into training and validation datasets with a 9:1 ratio).

Training Procedure

This model is primarily fine-tuned with LoRA.

Training Hyperparameters

• Training regime: fp16 mixed precision
• Training epochs: 10
• Batch size: 16
• Learning rate: 1e-3
• Max gradient norm: 1.0
• Weight decay: 0.0
• Lr scheduler: cosine
• Warmup ratio: 0.1

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was tested on the BlendX dataset ("dev" data).

Metrics

• Accuracy with micro
• F1 score with micro

Results

• Accuracy = 0.87
• F1 score = 0.93

Summary

Compute Infrastructure

This model was primarily trained using Google Colab (free tier).

Hardware

• GPU: NVIDIA T4 with 16GB VRAM
• RAM: 16GB
• Processor: Shared virtual CPUs (details depend on Google Colab free tier allocations)

Software

• PyTorch
• Jupyter Notebook
• Tensorboard

Model Card Contact

• twkang43@hanyang.ac.kr