Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
Paper • 2603.25562 • Published • 13
Papers LLM training tricks
Note Tricks to make On-Policy Distillation better
-
Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
Paper • 2604.13016 • Published • 77 -
From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space
Paper • 2604.14142 • Published • 26
TIP: Token Importance in On-Policy Distillation
Paper • 2604.14084 • Published • 11Note Informative tokens come from two regions: positions with high student entropy, and positions with low student entropy plus high teacher–student divergence, where the student is overconfident and wrong