Peer Collaborative Learning for Online Knowledge Distillation

By Guile Wu, Shaogang Gong. 2020

159 citations recorded by the source corpus.

Traditional knowledge distillation uses a two-stage training strategy to transfer knowledge from a high-capacity teacher model to a compact student model, which relies heavily on the pre-trained teacher. Recent online knowledge distillation alleviates this limitation by collaborative learning, mutual learning and online ensembling, following a one-stage end-to-end training fashion. However, collaborative learning and mutual learning fail to construct an online high-capacity teacher, whilst online ensembling ignores the collaboration among branches and its logit summation impedes the further optimisation of the ensemble teacher. In this work, we propose a novel Peer Collaborative Learning method for online knowledge distillation, which integrates online ensembling and network collaboration into a unified framework. Specifically, given a target network, we construct a multi-branch network for training, in which each branch is called a peer. We perform random augmentation multiple times on the inputs to peers and assemble feature representations outputted from peers with an additional classifier as the peer ensemble teacher. This helps to transfer knowledge from a high-capacity teacher to peers, and in turn further optimises the ensemble teacher. Meanwhile, we employ the temporal mean model of each peer as the peer mean teacher to collaboratively transfer knowledge among peers, which helps each peer to learn richer knowledge and facilitates to optimise a more stable model with better generalisation. Extensive experiments on CIFAR-10, CIFAR-100 and ImageNet show that the proposed method significantly improves the generalisation of various backbone networks and outperforms the state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Lan Du
6. Liyuan Sun
7. Shaogang Gong
8. Pan Li

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Shaogang Gong
3. Jianping Gou
4. S. Maybank
5. Xiaogang Wang
6. Baosheng Yu
7. Luc Van Gool
8. Pan Li

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “A variety of online knowledge distillation methods have been proposed, especially in the last few years (Zhang et al. 2018b; Chen et al. 2020a; Xie et al. 2019; Anil et al. 2018; Kim et al. 2019b; Zhou et al. 2018; Walawalkar et al. 2020; Wu and Gong 2021; Zhang et al. 2021a).”
• Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “Collaborative learning is the one used most often [25, 29, 32, 8, 9, 7], where the teacher and the student as peer networks collaboratively teach and learn from each other, and the peer network architectures can be different.”
• Striking a Balance between Stability and Plasticity for Class-Incremental Learning (2021, background): “To resolve this problem, many CIL studies [32, 26, 2] re-sort to storing some representative exemplars for rehearsal-based model learning and using a distillation loss [15, 40] for knowledge transfer.”