Large scale distributed neural network training through online distillation

By Rohan Anil, Gabriel Pereyra, Alexandre Passos, Róbert Ormándi, George E. Dahl, Geoffrey E. Hinton. 2018

455 citations recorded by the source corpus.

Techniques such as ensembling and distillation promise model quality improvements when paired with almost any base model. However, due to increased test-time cost (for ensembles) and increased complexity of the training pipeline (for distillation), these techniques are challenging to use in industrial settings. In this paper we explore a variant of distillation which is relatively straightforward to use as it does not require a complicated multi-stage setup or many new hyperparameters. Our first claim is that online distillation enables us to use extra parallelism to fit very large datasets about twice as fast. Crucially, we can still speed up training even after we have already reached the point at which additional parallelism provides no benefit for synchronous or asynchronous stochastic gradient descent. Two neural networks trained on disjoint subsets of the data can share knowledge by encouraging each model to agree with the predictions the other model would have made. These predictions can come from a stale version of the other model so they can be safely computed using weights that only rarely get transmitted. Our second claim is that online distillation is a cost-effective way to make the exact predictions of a model dramatically more reproducible. We support our claims using experiments on the Criteo Display Ad Challenge dataset, ImageNet, and the largest to-date dataset used for neural language modeling, containing $6\times 10^{11}$ tokens and based on the Common Crawl repository of web data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “In extreme cases, training must end before completing a single pass over the data (e.g. Anil et al., 2018).”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Finally, our work is also related to codistillation [1] where student and teacher have the same architecture and use distillation during training.”
Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, methodology): “In contrast, ensemble learning methods (You et al., 2017; Furlanello et al., 2018; Anil et al., 2018; Dvornik et al., 2019; Park & Kwak, 2019; Liu et al., 2019; Wu et al., 2019) allow to combine multiple heterogeneous weak classifiers by averaging the predictions of the individual models instead.”
Wireless Network Intelligence at the Edge (2018, background): “With these local and global logit vectors, the distillation regularizer ψ(F (i) k , F̂ (i) k ) is given as the gradient of their mean squared error [182] or of the cross entropy [118], [181].”
Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data (2018, methodology): “Instead, it requires exchanging either model outputs as many as the training dataset size, or model parameters so that the reproduced teacher model can locally generate outputs synchronously with the student model [11].”
Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, background): “Previous works only consider transferring knowledge from a large network to a smaller one [20, 21, 22, 23], or they transfer knowledge from a group, but each member in the group shares the same large model architecture or a large portion of the neural architecture with specific tail or head layers [”
Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer (2019, methodology): “This is the main objective of knowledge transfer [21], [9], [43], [44], [54], [7], which is a powerful tool to share knowledge of a model through its predictions.”
Diversity With Cooperation: Ensemble Methods for Few-Shot Classification (2019, methodology): “We enforce similarity on the full probability vectors in the ﬁrst one, computed with softmax at T = 10 following [1], and with conditionally non-ground-truth probabilities for the second one as deﬁned in Section 3.3.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “They then proposed a method named Augmented Box Replay to handle this problem, and a loss function named attentive Region-of-Interest distillation loss to further reduce catastrophic forgetting.”
Knowledge Distillation: A Survey (2020, methodology): “Anil et al. (2018) employed online distillation to train large-scale distributed neural network, and proposed a variant of online distillation called co-distillation.”
Multilingual Neural Machine Translation with Knowledge Distillation (2019, methodology): “Knowledge distillation has soon been applied to a variety of tasks, including image classification (Hinton et al., 2015; Furlanello et al., 2018; Yang et al., 2018; Anil et al., 2018; Li et al., 2017), speech recognition (Hinton et al.”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “Anil et al. (2018) introduced an efficient distributed online distillation framework called co-distillation and argue that distillation can even work when the teacher and student are made by the same network architecture.”
Knowledge Distillation by On-the-Fly Native Ensemble (2018, methodology): “[15] further extended this idea to accelerate large scale distributed neural network training.”
Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion (2019, background): “[4] ﬁrst applied knowledge distillation on neural networks, and then a lot of works expand the usage of knowledge distillation to a variety of tasks, such as image classiﬁcation [5, 24, 25] and natural language processing [6, 7, 8].”
Online Knowledge Distillation with Diverse Peers (2019, methodology): “Some approaches use individual networks with each one corresponding to a student model (Zhang et al. 2018; Anil et al. 2018), while some others ask all student models to share the same early blocks to further reduce the training cost (Song and Chai 2018; Lan, Zhu, and Gong 2018).”
Learning to Coordinate Multiple Reinforcement Learning Agents for Diverse Query Reformulation (2018, methodology): “Anil et al. (2018) later proposed a method to further reduce communication overhead by only exchanging the probability distributions of the different agents.”
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, methodology): “The knowledge can also be an ensemble of other student models’ logits [19], in which the ensemble of predictions is often more accurate than individual predictions.”
Revisiting Knowledge Distillation via Label Smoothing Regularization (2020, methodology): “Knowledge Distillation Since [7] proposed knowledge distillation based on prior work [2], KD has been widely adopted or modified [14, 19, 20, 4, 1, 11, 17].”
Gemma 3 Technical Report (2025, unknown): “Our post-training approach relies on an improved version of knowledge distillation (Agarwal et al., 2024; Anil et al., 2018; Hin-ton et al., 2015) from a large IT teacher, along with a RL finetuning phase based on improved versions of BOND (Sessa et al., 2024), WARM (Ramé et al., 2024b), and WARP…”