Self-Referenced Deep Learning

By Xu Lan, Xiatian Zhu, Shaogang Gong. 2018

27 citations recorded by the source corpus.

Knowledge distillation is an effective approach to transferring knowledge from a teacher neural network to a student target network for satisfying the low-memory and fast running requirements in practice use. Whilst being able to create stronger target networks compared to the vanilla non-teacher based learning strategy, this scheme needs to train additionally a large teacher model with expensive computational cost. In this work, we present a Self-Referenced Deep Learning (SRDL) strategy. Unlike both vanilla optimisation and existing knowledge distillation, SRDL distils the knowledge discovered by the in-training target model back to itself to regularise the subsequent learning procedure therefore eliminating the need for training a large teacher model. SRDL improves the model generalisation performance compared to vanilla learning and conventional knowledge distillation approaches with negligible extra computational cost. Extensive evaluations show that a variety of deep networks benefit from SRDL resulting in enhanced deployment performance on both coarse-grained object categorisation tasks (CIFAR10, CIFAR100, Tiny ImageNet, and ImageNet) and fine-grained person instance identification tasks (Market-1501).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. Xiatian Zhu
4. S. Maybank
5. Baosheng Yu
6. Mao Ye
7. Feng Zhang
8. Shifeng Zhang

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Xiatian Zhu
3. Yi Yang
4. Xiaogang Wang
5. Shaogang Gong
6. Luc Van Gool
7. Jianping Gou
8. Zhenguo Li

Citations in the corpus

• DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, background): “ing stopping Criterion 1*. Normal Cells Reduction Cells Figure 4: The cell of best structures searched on different datasets. Architecture Test Err. (%) Params (M) ResNet18 [38] 47:3 11:7 DenseNet-BC [17] 37:1 - NASNet 29:8 4:5 DARTS 30:4 3:8 DARTSy 46:1 2:1 SNAS 30:6 3:4 ASAP 30:0 3:3 DARTS+ 29:1 4”
• Knowledge Distillation: A Survey (2020, methodology): “In online setting, the teacher networks are usually ensembles of student networks, in which the student models share similar structure (or the same structure) with each other (Zhang et al. 2018b; Zhu and Gong 2018; Furlanello et al. 2018; Chen et al. 2020a).”
• Knowledge Distillation by On-the-Fly Native Ensemble (2018, background): “Recently, knowledge distillation has been exploited to distil easy-to-train large networks into harderto-train small networks [15], to transfer knowledge within the same network [20; 21], and to transfer high-level semantics across layers [8].”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, unknown): “To tackle these challenges, several novel self-distillation frameworks [24], [29], [79], [162], [163], [164], [165], [166], [167], [168], [169], [170] have been proposed recently.”
• Fast Human Pose Estimation (2018, background): “This principle has also been recently applied to accelerate the model training process of large scale distributed neural networks [2], to transfer knowledge between multiple layers [17] or between multiple training states [18].”