Collaborative Learning for Deep Neural Networks

By Guocong Song, Wei Chai. 2018

232 citations recorded by the source corpus.

We introduce collaborative learning in which multiple classifier heads of the same network are simultaneously trained on the same training data to improve generalization and robustness to label noise with no extra inference cost. It acquires the strengths from auxiliary training, multi-task learning and knowledge distillation. There are two important mechanisms involved in collaborative learning. First, the consensus of multiple views from different classifier heads on the same example provides supplementary information as well as regularization to each classifier, thereby improving generalization. Second, intermediate-level representation (ILR) sharing with backpropagation rescaling aggregates the gradient flows from all heads, which not only reduces training computational complexity, but also facilitates supervision to the shared layers. The empirical results on CIFAR and ImageNet datasets demonstrate that deep neural networks learned as a group in a collaborative way significantly reduce the generalization error and increase the robustness to label noise.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, background): “Previous works only consider transferring knowledge from a large network to a smaller one [20, 21, 22, 23], or they transfer knowledge from a group, but each member in the group shares the same large model architecture or a large portion of the neural architecture with specific tail or head layers [”
Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer (2019, background): “Furthermore, knowledge transfer acts as a strong regularization method and mitigates the risk of membership inference attacks [21], [49].”
Online Knowledge Distillation with Diverse Peers (2019, methodology): “Together with Table 2 and Figure 4, we can ﬁnd that the compared method CL-ILR and ONE improve the accuracy of individual student models ranging from 0.7% to 8% while decreasing the diversity among different peers, which sacriﬁces the effectiveness of ensemble models.”
UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation (2019, background): “The embedded design of UNet++ makes it amenable to auxiliary training, multi-task learning, and knowledge distillation [17], [37], [38].”
Feature Fusion for Online Mutual Knowledge Distillation (2019, methodology): “ways. This scheme was ﬁrst applied in an ofﬂine manner [3]. After that, an online KT was developed to enhance the performance of the student network which learns without a pre-trained teacher network [32,20,27,1]. This online learning method is related to our work in this paper. Ofﬂine KT is a way o”
Online Knowledge Distillation via Collaborative Learning (2020, methodology): “CLNN [22] promotes the diversity of each branch by a hierarchical multiple branch design and proposes to scale the gradient accordingly.”
Collaborative Learning for Faster StyleGAN Embedding (2020, background): “One is to simultaneously train multi-head models and then merge their features to obtain more useful features [35, 34].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “On the other aspect, [12], [28], [51], [57], [114], [118], [161], [174], [175], [182], [191], [201], [208], [210], [218], [231], [248], [258], [267] propose to add more teachers or other auxiliaries such as teaching assistant and small students to improve the robustness of ensemble distribution.”
Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, background): “Previous works only consider transferring knowledge from a large network to a smaller one [19, 20, 21, 22], or they transfer knowledge from a group, but each member in the group shares the same large model architecture or a large portion of the neural architecture with specific tail or head layers [”
Knowledge Transfer Graph for Deep Collaborative Learning (2019, background): “However, it is known that better accuracy can be achieved if learning is performed using multiple networks that can transfer knowledge among themselves [6, 15, 21, 17, 9].”
Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification (2020, methodology): “To enhance the discriminability and stabilize the graph attention learning, we employ a multi-head attention technique [38] by learning multiple h(·) and w (l = 1, 2 · · · , L, L is the total number of heads) with the same structure and optimizing them separately.”
When Machine Learning Meets Privacy (2020, background): “Although some collaborative learning models consider shared training data [145], which presents a significant privacy risk.”
The future of digital health with federated learning (2020, background): “There are also other collaborative learning strategies [47, 48] such as ensembling, a statistical strategy of”
Improved Techniques for Training Adaptive Deep Networks (2019, background): “to resolve the conﬂict while jointly optimizing all the classiﬁers. It has been observed in [14] that the individual classiﬁers in the network tend to negatively affect the learning of one other, and [31] discussed that the backbone network may not converge well due to 1 arXiv:1908.06294v1 [cs.CV] 1”
Peer Collaborative Learning for Online Knowledge Distillation (2020, methodology): “Instead of pre-training a high-capacity teacher, online distillation typically integrates the teacher into the student model using a hierarchical network with shared intermediate-level representations [21] (Fig.”
Knowledge Transfer via Dense Cross-Layer Mutual-Distillation (2020, methodology): “visual recognition task can transfer their current knowledge learnt on dierent sources of data to each other. [12] addressed multi-task machine translation problem with a dual co-training mechanism. [41] considered the co-training of several classier heads of the same network. Unlike these methods, ”
Hydra: Preserving Ensemble Diversity for Model Distillation (2020, methodology): “As discussed in detail by Song & Chai (2018), there exist multiple strategies to deﬁne a body of shared parameters together with heads , e.g. hierarchical sharing pattern.”
iDLG: Improved Deep Leakage from Gradients (2020, background): “In multi-node distributed learning systems such as Collaborative Learning [2, 3, 4] and Federated Learning [5, 6, 7], it is widely believed that sharing gradients between nodes will not leak the private training data.”