Model Card for FLAN-T5 Multitask SVD Adapter
Model Details
Model Description
This model is a multitask fine-tuned version of google/flan-t5-base that can perform both:
- Text Classification
- Text Summarization
It is built using parameter-efficient fine-tuning (LoRA, QLoRA, DoRA) and combines multiple adapters into a single unified adapter using Singular Value Decomposition (SVD).
- Developed by: Yenugu Sujithreddy
- Funded by: Self / Academic Project
- Shared by: Yenugu Sujithreddy
- Model type: Seq2Seq Transformer
- Language(s): English
- License: Apache-2.0
- Finetuned from model: google/flan-t5-base
Model Sources
- Repository: https://huggingface.co/your-username/flan-t5-multitask-svd
- Paper: N/A (Project-based implementation)
- Demo: N/A
Uses
Direct Use
This model can be used directly with prompt-based instructions:
Classification Example:
Task: classify
Text: Apple launches a new product
Summarization Example:
Task: summarize
Text: Long paragraph...
Downstream Use
- Chatbots with multitask capability
- News classification systems
- Content summarization tools
- AI assistants
Out-of-Scope Use
- Medical or legal decision-making
- Safety-critical systems
- Real-time high-risk applications
- Tasks without clear prompt instructions
Bias, Risks, and Limitations
- May inherit biases from training datasets
- Performance may degrade slightly after SVD merging
- Requires explicit task prompting
- Summaries may sometimes copy input text
- Limited generalization beyond trained domains
Recommendations
- Always validate outputs before use
- Use clear prompts (
Task: classify/Task: summarize) - Fine-tune further for production-grade applications
- Evaluate on domain-specific datasets
How to Get Started with the Model
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
model = AutoModelForSeq2SeqLM.from_pretrained("your-username/flan-t5-multitask-svd")
tokenizer = AutoTokenizer.from_pretrained("your-username/flan-t5-multitask-svd")
input_text = "Task: classify\nText: Tesla stock rises"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training Details
Training Data
- Custom classification dataset (news categories like Sports, Business, Science)
- Custom summarization dataset (text-summary pairs)
Training Procedure
Preprocessing
- Text cleaning
- Prompt formatting (task-based input)
- Tokenization using Flan-T5 tokenizer
Training Hyperparameters
- Training regime: fp16 mixed precision
- Learning rate: 2e-4
- Batch size: 8
- LoRA rank: 8
- Optimizer: AdamW
Evaluation
Testing Data, Factors & Metrics
Testing Data
- Held-out classification samples
- Sample summarization inputs
Factors
- Task type (classification vs summarization)
- Input length
- Domain variation
Metrics
- Classification: Accuracy
- Summarization: Qualitative + ROUGE (manual check)
Results
- Good classification accuracy on known categories
- Summarization produces concise outputs
- Slight performance trade-off due to adapter merging
Summary
The model successfully performs multitask learning using a single adapter, demonstrating the effectiveness of SVD-based adapter merging.
Model Examination
- No formal interpretability methods applied
- Behavior verified through manual testing
Environmental Impact
- Hardware Type: GPU
- Hours used: ~4–6 hours
- Cloud Provider: Local / Colab
- Compute Region: N/A
- Carbon Emitted: Not measured
Technical Specifications
Model Architecture and Objective
- Base: FLAN-T5
- Adapter Type: LoRA
- Merge Method: SVD
- Objective: Multitask learning (classification + summarization)
Compute Infrastructure
Hardware
- NVIDIA GPU (recommended)
Software
- PyTorch
- Hugging Face Transformers
- PEFT
Citation
BibTeX:
@misc{flant5_multitask_svd_2026,
title={FLAN-T5 Multitask Model using SVD Adapter Merging},
author={Yenugu Sujithreddy},
year={2026}
}
APA: Sujithreddy, Y. (2026). FLAN-T5 Multitask Model using SVD Adapter Merging.
Glossary
- LoRA: Low-Rank Adaptation
- QLoRA: Quantized LoRA
- SVD: Singular Value Decomposition
- Multitask Learning: Training a model for multiple tasks
More Information
This project demonstrates an advanced fine-tuning pipeline combining multiple PEFT techniques and adapter merging.
Model Card Authors
Yenugu Sujithreddy
Model Card Contact
For questions or collaboration, please open an issue on the Hugging Face repository.