A statistical perspective on distillation

107 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, positive): “We here build upon the seminal work of [24], which has recently proposed a statistical background of Knowledge Distillation (KD) helping researchers and practitioners to gain novel insights on its effectiveness.”
Large Language Models Are Reasoning Teachers (2022, methodology): “We note that this scaling effect is in contrast not a given in knowledge distillation, where more accurate teachers do not always result in better students (Menon et al., 2021).”