Big Neural Networks Waste Capacity

By Yann Dauphin, Yoshua Bengio. 2013

87 citations recorded by the source corpus.

This article exposes the failure of some big neural networks to leverage added capacity to reduce underfitting. Past research suggest diminishing returns when increasing the size of neural networks. Our experiments on ImageNet LSVRC-2010 show that this may be due to the fact there are highly diminishing returns for capacity in terms of training error, leading to underfitting. This suggests that the optimization method - first order gradient descent - fails at this regime. Directly attacking this problem, either through the optimization method or the choices of parametrization, may allow to improve the generalization error on large datasets, for which a large capacity is required.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, background): “Indeed, many explanations for the difficulty of deep learning have been advanced in the literature, including the presence of many local minima, low curvature regions due to saturating nonlinearities, and exponential growth or decay of back-propagated gradients [12, 13, 14, 15].”
Measuring the Intrinsic Dimension of Objective Landscapes (2018, background): “( i ) While it has previously been appreciated that large networks waste parameters (Dauphin & Bengio, 2013) and weights contain redundancy (Denil et al., 2013) that can be exploited for post-hoc compression (Wen et al., 2016), this paper’s method constitutes a much simpler approach to compression,…”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, methodology): “[15] trains a shallow network on SIFT features to classify the ImageNet dataset.”
Do Deep Nets Really Need to be Deep? (2013, background): “On the TIMIT phoneme recognition and CIFAR-10 image recognition tasks, shallow nets can be trained that perform similarly to complex, well-engineered, deeper convolutional architectures.”
Deep Learning of Representations: Looking Forward (2013, background): “As we will see in Section 4, we hypothesize that as the size of the models increases, our current ways of training deep networks become less and less efficient, so that the computation required to train larger models (to capture correspondingly more information) is likely to scale much worse than li”
Knowledge Matters: Importance of Prior Information for Optimization (2013, background): “It has been observed that the condition number of the Hessian matrix (speciﬁcally, its largest eigenvalue) increases as the size of the network increases (Dauphin and Bengio, 2013), making training considerably slower and ineﬃcient (Dauphin and Bengio, 2013).”
Do Deep Convolutional Nets Really Need to be Deep and Convolutional? (2016, background): “Dauphin and Bengio (2013) trained shallow nets on SIFT features to classify a large-scale ImageNet dataset and found that it was difficult to train large, high-accuracy, shallow nets.”
Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture (2020, background): “This is commonly seen as an over-parameterization (Dauphin and Bengio, 2013; Denil et al., 2013).”
Data augmentation instead of explicit regularization (2018, background): “Hence, it seems that with these standard regularization methods deep networks are wasting capacity (Dauphin and Bengio, 2013).”
Sequential Covariance-Matrix Estimation with Application to Mitigating Catastrophic Forgetting (2015, background): “It was shown in [1] that there is a law of diminishing returns for increasing the capacity of a network by adding more neurons.”
Training Neural Networks with Stochastic Hessian-Free Optimization (2013, background): “More recently, [16] argue that large neural networks waste capacity in the sense that adding additional units fail to reduce underﬁtting on large datasets.”
Tinier-YOLO: A Real-Time Object Detection Method for Constrained Environments (2020, methodology): “However, the expressiveness capability of shallow netowork is far away from the DNN models [38].”
Neuron Clustering for Mitigating Catastrophic Forgetting in Supervised and Reinforcement Learning (2015, background): “In [9] a law of diminishing returns is demonstrated, whereby adding capacity to the network ceases to contribute capturing of new representations.”
Blending LSTMs into CNNs (2015, background): “There is evidence that feedforward neural networks trained with current training algorithms use their large capacity inefﬁciently (Le Cun et al., 1990; Denil et al., 2013; Dauphin & Bengio, 2013; Ba & Caruana, 2014; Hinton et al., 2015; Han et al., 2016).”
Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask (2019, background): “Many neural networks are over-parameterized [3, 4], enabling compression of each layer [4, 21, 8] or of the entire network [14].”
Neuron clustering for mitigating catastrophic forgetting in feedforward neural networks (2014, background): “There appears to be a law of diminishing returns such that adding capacity the network captures fewer features of the large dataset [3].”