Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution
By Hadi Pouransari, Oncel Tuzel. 2020
5 citations recorded by the source corpus.
Knowledge distillation has been used to transfer knowledge learned by a sophisticated model (teacher) to a simpler model (student). This technique is widely used to compress model complexity. However, in most applications the compressed student model suffers from an accuracy gap with its teacher. We propose extracurricular learning, a novel knowledge distillation method, that bridges this gap by (1) modeling student and teacher output distributions; (2) sampling examples from an approximation to the underlying data distribution; and (3) matching student and teacher output distributions over this extended set including uncertain samples. We conduct rigorous evaluations on regression and classification tasks and show that compared to the standard knowledge distillation, extracurricular learning reduces the gap by 46% to 68%. This leads to major accuracy improvements compared to the empirical risk minimization-based training for various recent neural network architectures: 16% regression error reduction on the MPIIGaze dataset, +3.4% to +9.1% improvement in top-1 classification accuracy on the CIFAR100 dataset, and +2.9% top-1 improvement on the ImageNet dataset.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohammad Rastegari
- Sachin Mehta
- Oncel Tuzel
- Hadi Pouransari
- Ali Farhadi
- Mehrdad Farajtabar
- Fartash Faghri
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Mohammad Rastegari
- Sachin Mehta
- Oncel Tuzel
- Luc Van Gool
- Xiaogang Wang
- Ali Farhadi
- Dacheng Tao
- Ming-Hsuan Yang
In Representation & Self-Supervised Learning
- Ali Farhadi
- Sachin Mehta
- Mehrdad Farajtabar
- Hadi Pouransari
- Mohammad Rastegari
- Oncel Tuzel
- Fartash Faghri
Citations in the corpus
- Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, background): “Distilling feature maps and probability distributions between the random pair of original images and their MixUp images was proposed to guide the network to learn cross-image knowledge [46, 66].”