Do Deep Nets Really Need to be Deep?

By Jimmy Ba, R. Caruana. 2013

2,250 citations recorded by the source corpus.

Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this paper we empirically demonstrate that shallow feed-forward nets can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow nets can learn these deep functions using the same number of parameters as the original deep models. On the TIMIT phoneme recognition and CIFAR-10 image recognition tasks, shallow nets can be trained that perform similarly to complex, well-engineered, deeper convolutional models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Signal Processing

Citations in the corpus

FitNets: Hints for Thin Deep Nets (2014, methodology): “The idea has been recently adopted in Ba & Caruana (2014) to compress deep and wide networks into shallower but even wider ones, where the compressed model mimics the function learned by the complex model, in this case, by using data labeled by a deep (or an ensemble of deep) networks.”
Sequence-Level Knowledge Distillation (2016, methodology): “For classiﬁcation this usually means matching the probabilities either via L 2 on the log scale (Ba and Caruana, 2014) or by cross-entropy (Li et al., 2014; Hinton et al., 2015).”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, methodology): “Motivated by these results, we propose a way of applying knowledge distillation (Bucilu et al., 2006; Ba and Caruana, 2014; Hinton et al., 2015) so that single-task models effectively teach a multi-task model.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “ potential of exploring BERT variants. Shifting topics, knowledge distillation refers a general set of techniques where a smaller student model learns to mimic the behavior of a larger teacher model [Ba and Caruana, 2014, Hinton et al., 2015]. Generally, the goal is for the student model to achieve ”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “As we use soft targets, our work is also related to methods in Knowledge Distillation [7, 3, 26, 16].”
Large Language Models Can Self-Improve (2022, neutral): “Our method tangentially relates to rich literature on distillation (Ba and Caruana, 2014; Hinton et al., 2015), where a student network imitates a teacher network’s classifier predictions on input examples.”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “We build on top of the well-known Knowledge Distilla-tion framework [9, 2, 35], the effectiveness of which has been extensively studied [12, 56].”
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, methodology): “Efforts in this area could be broadly categorized as follows: (i) new architectures that are compact by design [11, 13]; (ii) automated neural architecture search (NAS) with reward function set as latency or model size [32, 35]; (iii) pruning based methods to reduce model size of existing architectu”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “olving pruning. Our contribution to this class of approaches is to demonstrate that small, trainable networks exist within larger networks in the form of winning tickets. After training. Distillation [2, 21] trains small networks to mimic the behavior of large networks; small networks are easier to ”
Born Again Neural Networks (2018, methodology): “In both cases, the idea is typically to transfer the knowledge of a high-capacity teacher with formidable performance to a more compact student [8, 9, 10].”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, background): “[16] provides empirical evidence on small datasets ( e.g., CIFAR-10) that shallow nets are capable of learning the same functions as deep nets.”
Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning (2015, background): “These output targets are either the ﬁnal layer logits (Ba & Caruana, 2014) or the high-temperature softmax outputs of the experts (Hinton et al., 2015).”
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks (2019, methodology): “In particular, Ba and Caruana (2014) posit that, in addition to a one-hot predicted label, the teacher’s predicted probability is also important.”
SoundNet: Learning Sound Representations from Unlabeled Video (2016, methodology): “Our method builds upon teacher-student models [3, 9] and dark knowledge transfer [13].”
Compressing Neural Networks with the Hashing Trick (2015, methodology): “Speciﬁcally, Hinton et al. (2014) and Ba & Caruana (2014) train a large network on the original training labels, then learn a much smaller “dis-tilled” model on a weighted combination of the original labels and the (softened) softmax output of the larger model.”
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks (2015, background): “The intuition was suggested in [18] while distillation itself was formally introduced in [19].”
Wireless Network Intelligence at the Edge (2018, background): “With these local and global logit vectors, the distillation regularizer ψ(F (i) k , F̂ (i) k ) is given as the gradient of their mean squared error [182] or of the cross entropy [118], [181].”
Practical Black-Box Attacks against Machine Learning (2016, background): “ork. Finally, we assumed attackers only had access to the label O˜ assigned by the oracle. Attackers with access to the probability vector O could more accurately extract the oracle model’s knowledge [2, 19]. However, this attack is outside of the scope of this paper. In some cases, it may be difﬁcu”