Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space

140 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Knowledge Distillation: A Survey (2020, neutral): “69 ↑) AE-KD (Du et al., 2020) FeaK ResNet56 (—) MobileNetV2 (75.”
MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks (2020, unknown): “riority and learning ability, no matter what kind of structures on teacher and student you use. It can be observed that our gaps between teachers and students are smaller than both CRD [48] and AE-KD [7] by a signiﬁcant margin. (2) the absolute accuracy on the same structure of student. If choosing ”
Knowledge Distillation from A Stronger Teacher (2022, background): “Besides this vanilla prediction match, other works [11, 14, 34, 41] also investigate the knowledge within intermediate representations to further boost the distillation performance, but this usually induces additional training cost as a consequence.”
Distilling Object Detectors via Decoupled Features (2021, methodology): “Since then, KD has been widely adopted in classification tasks [22, 63, 3, 54, 7, 65, 10].”