Born Again Neural Networks

By Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, Anima Anandkumar. 2018

1,252 citations recorded by the source corpus.

Knowledge distillation (KD) consists of transferring knowledge from one machine learning model (the teacher}) to another (the student). Commonly, the teacher is a high-capacity model with formidable performance, while the student is more compact. By transferring knowledge, one hopes to benefit from the student's compactness. %we desire a compact model with performance close to the teacher's. We study KD from a new perspective: rather than compressing models, we train students parameterized identically to their teachers. Surprisingly, these {Born-Again Networks (BANs), outperform their teachers significantly, both on computer vision and language modeling tasks. Our experiments with BANs based on DenseNets demonstrate state-of-the-art performance on the CIFAR-10 (3.5%) and CIFAR-100 (15.5%) datasets, by validation error. Additional experiments explore two distillation objectives: (i) Confidence-Weighted by Teacher Max (CWTM) and (ii) Dark Knowledge with Permuted Predictions (DKPP). Both methods elucidate the essential components of KD, demonstrating a role of the teacher outputs on both predicted and non-predicted classes. We present experiments with students of various capacities, focusing on the under-explored case where students overpower teachers. Our experiments show significant advantages from transferring knowledge between DenseNets and ResNets in either direction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, background): “This ﬁnding is consistent with research on knowledge distillation (Hinton et al., 2015; Furlanello et al., 2018) where soft targets also work notably better than hard targets.”
Technical report on Conversational Question Answering (2019, background): “Knowledge Distillation(KD) (Furlanello et al., 2018) transfer ”knowledge” from one machine learning model (teacher) to another (student) by using teacher’s output as student’s training objec-tive.”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “Clark et al. (2019b) demonstrate that knowledge distillation (Hinton et al., 2015; Furlanello et al., 2018) can lead to student networks that outperform their teachers.”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “In “born-again networks” (Furlanello et al., 2018), the teacher and student have the same neural architecture and model size, but surprisingly the student is able to surpass the teacher’s accuracy.”
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “We exploit the Born-again [13] strategy to apply KD sequentially to generate multiple generations, which is shown in Figure 3.”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, methodology): “While most methods focus on distilling knowledge from a pre-trained teacher model [28, 29, 30, 31, 32], online distillation [33, 34] simultaneously trains multiple models and use their ensemble as the teacher.”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “Numerous variants of KD have been proposed, including feature distillation [28, 73], iterative distillation [43, 67], and self-distillation [65, 44, 18, 29].”
Blockwise Parallel Decoding for Deep Autoregressive Models (2018, neutral): “…distillation (Hinton et al., 2015; Kim and Rush, 2016), in which one trains a model on the outputs of another model, has been shown to improve performance on a variety of tasks, potentially even when teacher and student models have the same architecture and model size (Furlanello et al., 2017).”
Experience Replay for Continual Learning (2018, background): “Reducing this deviation is our motivation for the behavioral cloning component of CLEAR, inspired by work showing the power of policy consolidation [12], self-imitation [23], and knowledge distillation [5].”
FEED: Feature-level Ensemble for Knowledge Distillation (2019, methodology): “This has a thread of connection with that of BAN (Furlanello et al. 2018), which shows that using the same architecture for both the student and the teacher is actually beneficial.”
Zero-Shot Knowledge Distillation in Deep Networks (2019, background): “In (Furlanello et al., 2018), knowledge transfer is done across several generations where the student of current generation learns from its previous generation.”
Variational Information Distillation for Knowledge Transfer (2019, background): “[9], the student network with the same size as the teacher network outperforms the teacher network with all the knowledge transfer methods.”
DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, background): “The soft-annotation is supposed to help the student to learn faster than the teacher did [Furlanello et al., 2018].”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, positive): “• Dark Experience Replay (DER++) [15]: the authors exploit self-distillation [28] to take advantage of the previously learned knowledge.”
Transfer without Forgetting (2022, background): “– Dark Experience Replay (DER) [8] envisions a self-distillation [22] constraint on data stored in the memory buffer and represents a simple extension to the basic rehearsal strategy of ER.”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, background): “Various solutions have been proposed to enhance its effectiveness, such as response-based distillation [3], feature-based distillation [8] and relation-based distillation [6].”
Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022, methodology): “Besides, Furlanello et al. (2018) performed iterative selfdistillation where the student network is identical to the teacher in terms of the network graph.”