Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification

By Liuyu Xiang, Guiguang Ding. 2020

345 citations recorded by the source corpus.

In real-world scenarios, data tends to exhibit a long-tailed distribution, which increases the difficulty of training deep networks. In this paper, we propose a novel self-paced knowledge distillation framework, termed Learning From Multiple Experts (LFME). Our method is inspired by the observation that networks trained on less imbalanced subsets of the distribution often yield better performances than their jointly-trained counterparts. We refer to these models as 'Experts', and the proposed LFME framework aggregates the knowledge from multiple 'Experts' to learn a unified student model. Specifically, the proposed framework involves two levels of adaptive learning schedules: Self-paced Expert Selection and Curriculum Instance Selection, so that the knowledge is adaptively transferred to the 'Student'. We conduct extensive experiments and demonstrate that our method is able to achieve superior performances compared to state-of-the-art methods. We also show that our method can be easily plugged into state-of-the-art long-tailed classification algorithms for further improvements.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiaya Jia
2. Shu Liu
3. Ziwei Liu
4. Zhisheng Zhong
5. Jiequan Cui
6. Andrew D. Bagdanov
7. Xialei Liu
8. Stella X. Yu

Citations in the corpus

• Long-Tailed Class Incremental Learning (2022, background): “Learning from Multiple Experts (LFME) [42] and Routing Diverse Experts (RIDE) [39] adopt the same idea of ensemble learning to aggregate knowledge from multiple experts.”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “To tackle these problems, some recent works [64], [182], [191], [219], [231], [234] propose data distillation by unifying data sources from multiple teachers as illustrated in Fig.”
• A Survey on Curriculum Learning (2021, methodology): “For example, LFME [125] jointly adopts an SPL-like mechanism for expert selection (each expert is trained on a subset of training data) in knowledge distillation and a Transfer Teacher for instance selection in each subset.”
• Improving Calibration for Long-Tailed Recognition (2020, methodology): “To verify the effectivity, we compare the proposed method against previous one-stage methods of Range Loss [38], LDAM Loss [4], FSLwF [8], and OLTR [20], and against previous two-stage methods, including DRS-like, DRW-like [4], LFME [35], cRT, and LWS [15].”
• Long-tailed Recognition by Routing Diverse Distribution-Aware Experts (2020, positive): “By using RIDE, cross-entropy loss (without any re-balancing strategies) can even outperforms current SOTA method LFME.”
• Balanced Knowledge Distillation for Long-tailed Learning (2021, methodology): “Indeed, without multiple teachers and complex learning schedules, our method outperforms LFME on all the long-tailed benchmarks we report by a large margin.”
• Curriculum Temperature for Knowledge Distillation (2022, background): “In knowledge distillation, LFME (Xiang, Ding, and Han 2020) adopt the classic curriculum strategy and propose to train the student gradually using samples ordered in an easy-to-hard sequence.”