Natural Neural Networks

By Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, Koray Kavukcuoglu. 2015

191 citations recorded by the source corpus.

We introduce Natural Neural Networks, a novel family of algorithms that speed up convergence by adapting their internal representation during training to improve conditioning of the Fisher matrix. In particular, we show a specific example that employs a simple and efficient reparametrization of the neural network weights by implicitly whitening the representation obtained at each layer, while preserving the feed-forward computation of the network. Such networks can be trained efficiently via the proposed Projected Natural Gradient Descent algorithm (PRONG), which amortizes the cost of these reparametrizations over many parameter updates and is closely related to the Mirror Descent online learning algorithm. We highlight the benefits of our method on both unsupervised and supervised learning tasks, and showcase its scalability by training on the large-scale ImageNet Challenge dataset.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Gaussian Error Linear Units (GELUs) (2016, methodology): “We found that it is possible to train competitive MNIST and TIMIT networks solely with this stochastic regularizer, all without using any nonlinearity.”
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, methodology): “…inverse can for example be obtained by using a Kronecker factored approximation to the Fisher matrix and inverting it (KFAC, [19]), by using an approximate Cholesky factorization of the inverse Fisher matrix (FANG, [8]), or by whitening the input of each layer in the neural network (PRONG, [3]).”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “As already noted by Desjardins et al. (2015), higher learning rates seem to perform better.”
Sharp Minima Can Generalize For Deep Nets (2017, methodology): “This approach has been used with success to improve optimization, by moving from a region of high curvature to a region of well behaved curvature (e.g. Desjardins et al., 2015; Salimans & Kingma, 2016).”
Concept whitening for interpretable image recognition (2020, background): “Earlier attempts whiten by periodically estimating the whitening matrix (Desjardins et al., 2015; Luo, 2017), which leads to instability in training.”
Incremental Object Detection via Meta-Learning (2020, positive): “A scalable way to achieve such a gradient preconditioning is to embed it directly into the task learner [33], [36], [37].”
Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting (2018, methodology): “Additionally, in many cases the required matrix is the inverse of the FIM (for example in NGD methods [13]), which is signiﬁcantly simpler and faster to compute for diagonal matrices.”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “It has been successfully applied to a variety of problems such as blind source separation (Amari and Cichocki, 1998), reinforcement learning (Peters and Schaal, 2008), and neural network training (e.g. Park et al., 2000; Martens and Grosse, 2015; Desjardins et al., 2015).”
Self-Normalizing Neural Networks (2017, background): “Natural neural networks [6] also aim at normalizing the variance of activations by reparametrization of the weights.”
Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks (2016, methodology): “However, Desjardins et al. (2015) suggest to overcome this problem by approximating this SVD by: a) using sub-sampled training data to compute this SVD; b) computing it every few number of iteration and relying on the assumption that this SVD approximately holds for the iterations in between. In add”
A Kronecker-factored approximate Fisher matrix for convolution layers (2016, methodology): “Projected Natural Gradient (PRONG; Desjardins et al., 2015) goes a step further than centering methods by fully whitening the activations in each layer.”
Batch normalized recurrent neural networks (2015, background): “Future directions include whitening input-to-hidden connections [10] and normalizing the hidden state instead of just a portion of the network.”
Learning values across many orders of magnitude (2016, methodology): “However, this assumes the data is available a priori, which is not true in online (potentially non-stationary) settings.”
Hessian-based Analysis of Large Batch Training and Robustness to Adversaries (2018, background): “There have been several similar attempts to change the optimization algorithm to find “better” regions [8, 5].”
Meta-Learning with Warped Gradient Descent (2019, methodology): “As noted by Desjardins et al. (2015), expressing preconditioning through interleaved projections can be seen as a form of Mirror Descent (Beck & Teboulle, 2003).”
Data-Dependent Path Normalization in Neural Networks (2015, methodology): “In this paper, we focus on two efficient alternative optimization approaches proposed recently for feed-forward neural networks that are based on intuitions about parametrization, normalization and the geometry of parameter space: Path-SGD (Neyshabur et al., 2015a) was derived as steepest descent…”
Blockout: Dynamic Model Selection for Hierarchical Deep Networks (2015, background): “through convolution or weight decay), rescaling or whitening internal representations for better conditioning [6, 10], or randomly perturbing activations for improved robustness and better generalizability [9, 26, 25].”
Out-of-Distribution Generalization via Risk Extrapolation (REx) (2020, background): “In particular, we note that increasing the relative weight of the penalty term after 100 epochs of training (using a so-called “waterfall” schedule (Desjardins et al., 2015)) is critically important to performance on the colored MNIST task, see Figure 4(b).”