Learning from Multiple Teacher Networks

By Shan You, Chao Xu, Dacheng Tao. 2017

464 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

FEED: Feature-level Ensemble for Knowledge Distillation (2019, background): “Shen(Shen et al. 2019) tried to combine knowledge trained from different domains, and You (You et al. 2017) utilized ensemble of orderings of samples to teach the student network, which is very novel.”
GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, background): “By dint of automatic feature engineering, deep neural networks (DNNs) have achieved remarkable success in various computer vision tasks, such as image classification [37, 36, 41, 33, 32, 15, 39], visual generation [34, 35], image retrieval [40, 7, 8, 12] and semantic comprehension [18, 17].”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “You et al. (2017) proposed a voting strategy to unify multiple relative dissimilarity information provided by multiple teacher networks.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “The choices of teacher models are also well studied, where [227, 265] use multiple teacher models to jointly supervise the student model, while [43, 269] apply self-distillation without an extra teacher model.”
Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks (2017, methodology): “Inspired by the ideas introduced by Zagoruyko and Komodakis [21], we show that in a constrained teacherstudent learning setting [22]–[25], knowledge transfer to a shallow student (commonly called knowledge distillation) is possible from the explanation of CNN decisions generated by Grad-CAM++.”
Few-shot learning of neural networks from scratch by pseudo example optimization (2018, background): “Other previous researches demonstrates that knowledge can be transferred from either intermediate layers of a teacher model [28, 37], from multiple teacher models [38], mismatched unsupervised stimuli [20] or examples reconstructed from meta-data [10].”
Learning With Single-Teacher Multi-Student (2018, methodology): “Different from our prior work (You et al. 2017b) aiming at “Multi-Teacher Single-Student” problem where multiple teachers have the same functions, our work highlights multiple students and they possess different functions from each other (i.e. covering different binary classes).”
Towards Evolutionary Compression (2018, methodology): “[47] utilized multiple teacher networks for training the compressed network.”
Stochasticity and Skip Connection Improve Knowledge Transfer (2021, background): “Recently, knowledge transfer using multiple teacher networks have been proposed [9].”
Distilling Object Detectors via Decoupled Features (2021, methodology): “Since then, KD has been widely adopted in classification tasks [22, 63, 3, 54, 7, 65, 10].”
Adversarial Learning of Portable Student Networks (2018, background): “Moreover, (You et al. 2017) simultaneously utilized multiple teacher networks for learning a more accurate student network.”
Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks (2017, background): “Another line of research focuses on transferring intermediate features instead of soft targets from teacher to student [30, 41, 47, 44, 15, 49, 45].”
Imitation networks: Few-shot learning of neural networks from scratch (2018, unknown): “…knowledge can be transferred from either intermediate layers of a teacher model [Romero et al., 2015; Yim et al., 2017], from multiple teacher models [You et al., 2017], mismatched unsupervised stimuli [Kulkarni et al., 2017] or examples reconstructed from meta-data [Gontijo Lopes et al., 2017].”
Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets (2020, background): “Knowledge distillation [13, 37, 55, 51] attempts to teach a compact model, also called student model, with knowledge distilled from a large teacher network.”
Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation (2019, background): “It contains two main research directions: cross-domain transfer learning [22, 12, 10, 4] and cross-task one [9, 3, 5, 35].”
Low-resolution Visual Recognition via Deep Feature Distillation (2019, background): “[22] proposed to combine the knowledge of multiple teacher networks in the intermediate representations to train a thin deep student network.”