Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net

By Guorui Zhou, Ying Fan, Runpeng Cui, Weijie Bian, Xiaoqiang Zhu, Kun Gai. 2017

129 citations recorded by the source corpus.

Models applied on real time response tasks, like click-through rate (CTR) prediction model, require high accuracy and rigorous response time. Therefore, top-performing deep models of high depth and complexity are not well suited for these applications with the limitations on the inference time. In order to get neural networks of better performance given the time limitations, we propose a universal framework that exploits a booster net to help train the lightweight net for prediction. We dub the whole process rocket launching, where the booster net is used to guide the learning of our light net throughout the whole training process. We analyze different loss functions aiming at pushing the light net to behave similarly to the booster net. Besides, we use one technique called gradient block to improve the performance of light net and booster net further. Experiments on benchmark datasets and real-life industrial advertisement data show the effectiveness of our proposed method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “Interestingly, the parameter sharing of intermediate layers of the teacher model together with response-based knowledge is also used as the teacher knowledge (Zhou et al. 2018).”
Improving the Interpretability of Deep Neural Networks with Knowledge Distillation (2018, background): “Afterwards, a number of works followed such as [34] [35] [36], just to name a few.”
Knowledge Distillation via Instance Relationship Graph (2019, methodology): “[30] made the student share some lower layers with the teacher and train them simultaneously, but they also used logits as the distilled knowledge.”
Knowledge Distillation via Route Constrained Optimization (2019, methodology): “From the perspective of optimization route, we find method in [33] could be regarded as a particular case of RCO when setting Mgap to the smallest value, and matching logits with KD loss instead of MSE loss.”
Lifelong Teacher-Student Network Learning (2021, background): “While such approaches provide a good performance [35], [36], they are unable to preserve well previously acquired information and related tasks.”
Learning Compatible Embeddings (2021, background): “Various knowledge sources are adopted in literature such as classification logits [17, 1, 28], hint layers [35, 54, 49, 3] and multi-layer groups [52, 51, 20].”
Knowledge distillation via adaptive instance normalization (2020, methodology): “Zhou et al. [17] proposed a novel method which introduces a booster net which is trained together with the student network and a logit matching loss for supervising the student’s prediction.”
Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation (2019, unknown): “Comparing to the traditional neural network compression methods [4] which focus on compressing the original network directly, a solution with the knowledge distillation to compress the deep network attracts more attention from research community in recent years, such as in the tasks of image recogni”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “Rocket launching [187] is a parallel learning scheme that trains both the teacher and student models simultaneously.”
Low-Resolution Face Recognition in the Wild via Selective Knowledge Distillation (2018, methodology): “With the development of much deeper and wider networks, such distillation technique has been adopted in many works [38]–[49] to compress a complex model (or an ensemble) into a simpler model that is much easier to deploy.”
Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks (2017, background): “Another line of research focuses on transferring intermediate features instead of soft targets from teacher to student [30, 41, 47, 44, 15, 49, 45].”
Knowledge Squeezed Adversarial Network Compression (2019, result): “While our approach is complementary to the above result-oriented learning methods by introducing the process-oriented learning strategy, which pays more attention on transferring intermediate knowledge [34, 35, 36, 37] instead of final result from teacher to student.”