The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks
By George Philipp, Jaime G. Carbonell. 2018
14 citations recorded by the source corpus.
For a long time, designing neural architectures that exhibit high performance was considered a dark art that required expert hand-tuning. One of the few well-known guidelines for architecture design is the avoidance of exploding gradients, though even this guideline has remained relatively vague and circumstantial. We introduce the nonlinearity coefficient (NLC), a measurement of the complexity of the function computed by a neural network that is based on the magnitude of the gradient. Via an extensive empirical study, we show that the NLC is a powerful predictor of test error and that attaining a right-sized NLC is essential for optimal performance. The NLC exhibits a range of intriguing and important properties. It is closely tied to the amount of information gained from computing a single network gradient. It is tied to the error incurred when replacing the nonlinearity operations in the network with linear operations. It is not susceptible to the confounders of multiplicative scaling, additive bias and layer width. It is stable from layer to layer. Hence, we argue that the NLC is the first robust predictor of overfitting in deep networks.
Subfields: Artificial Intelligence, Computational Mechanics, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Greg Yang
- Sam Schoenholz
- Jeffrey Pennington
- Yoshua Bengio
- Jascha Narain Sohl-Dickstein
- Yann Dauphin
- Quoc V. Le
- Mohit Bansal
In Computational Mechanics
- Greg Yang
- Jeffrey Pennington
- Sam Schoenholz
- Jascha Narain Sohl-Dickstein
- Yann Dauphin
- Boris Hanin
- Chunyuan Li
In Scaling & Learning Theory
- Greg Yang
- Sam Schoenholz
- Jeffrey Pennington
- Jascha Narain Sohl-Dickstein
- Yann Dauphin
- Vinay Rao
- Lechao Xiao
- Boris Hanin
Citations in the corpus
- A Mean Field Theory of Batch Normalization (2019, background): “We remark that Philipp & Carbonell (2018) has also empirically noted gradient explosion happens in deep batchnorm networks with various nonlinearities.”
- MetaInit: Initializing learning by learning to initialize (2019, background): “[47] who showed correlation between generalization and “nonlinearity”.”