Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
Paper • 2402.12030 • Published • 3
ULD Loss (Universal LLMs Distillation)
The ULD loss, based on optimal transport, enables distillation across different LLM families without requiring shared tokenizers.
-
-
mistralai/Mistral-7B-Instruct-v0.2
Text Generation • 7B • Updated • 2.01M • • 3.13k -
meta-llama/Llama-2-7b-chat-hf
Text Generation • Updated • 422k • 4.74k -
EleutherAI/pythia-160m-deduped
Text Generation • 0.2B • Updated • 93.5k • 3 -
EleutherAI/pythia-410m-deduped
Text Generation • 0.5B • Updated • 21.4k • 20 -
EleutherAI/pythia-1b-deduped
Text Generation • Updated • 10.1k • 19 -
bigscience/bloomz-560m
Text Generation • 0.6B • Updated • 1.35M • 137 -
bigscience/mt0-base
Text Generation • 0.6B • Updated • 4.46k • 32 -
facebook/opt-350m
Text Generation • Updated • 160k • 149 -
rajpurkar/squad
Viewer • Updated • 98.2k • 135k • 360 -
google-research-datasets/qed
Updated • 264 • 3 -
GEM/FairytaleQA
Viewer • Updated • 10.6k • 167 • 9 -
qiaojin/PubMedQA
Viewer • Updated • 274k • 22.9k • 317 -
knkarthick/dialogsum
Viewer • Updated • 14.5k • 9.14k • 240