Attentive Student Meets Multi-Task Teacher: Improved Knowledge Distillation for Pretrained Models

18 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “Self-supervised Pretraining (MLM) We closely follow the pretraining setup and model from R O BERT A (Liu et al., 2019b) with a few minor exceptions.”
Poor Man's BERT: Smaller and Faster Transformer Models (2020, methodology): “Others attempted to distill task-speciﬁc BERT for NLU tasks (Jiao et al., 2019), reﬁned by multi-task learning on the GLUE benchmark (Liu et al., 2019a).”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “In such a setting, the student is encouraged to learn the softened output of the assembled teachers’ logits (dark knowledge) via the cross-entropy loss as done in [9], [51], [57], [101], [114], [134], [139], [155], [161], [174], [208], [210], [214], [232], [249], [258], [272], which can be generaliz”