Riemannian metrics for neural networks I: feedforward networks

By Yann Ollivier. 2013

106 citations recorded by the source corpus.

We describe four algorithms for neural network training, each adapted to different scalability constraints. These algorithms are mathematically principled and invariant under a number of transformations in data and network representation, from which performance is thus independent. These algorithms are obtained from the setting of differential geometry, and are based on either the natural gradient using the Fisher information matrix, or on Hessian methods, scaled down in a specific way to allow for scalability while keeping some of their key mathematical properties.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

Citations in the corpus

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “LR 2016 The next Theorem 1 gives the correction of the standard gradient by the unit natural gradient where the bias weight is treated separately. The resulting formulas are similar to Deﬁnition 6 of Olivier (2013). Theorem 1. The unit natural gradient corrects the weight update to a unit ( wT ; w 0”
Robust Large Margin Deep Neural Networks (2016, background): “Reparametrization of DNN for more efficient learning is studied in depth in [21].”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “This problem can be addressed through the use of various approximations to the Fisher (e.g Le Roux et al., 2008; Ollivier, 2015; Grosse and Salakhudinov, 2015; Martens and Grosse, 2015) that are designed to be easier to compute, to store and finally to invert, than the exact Fisher.”
A Kronecker-factored approximate Fisher matrix for convolution layers (2016, background): “Approximations to natural gradient don’t satisfy full invariance to parameterization, but certain approximations have been shown to be invariant to more limited, but still fairly broad, classes of transformations (Ollivier, 2015; Martens & Grosse, 2015).”
From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification (2016, background): “…model (Bradley & Terry, 1952; Zadrozny, 2001; Menke & Martinez, 2008), the multinomial probit (Albert & Chib, 1993), the spherical softmax (Ollivier, 2013; Vincent, 2015; de Br´ebisson & Vincent, 2015), or softmax approximations (Bouchard, 2007), while theoretically or computationally…”
Understanding symmetries in deep networks (2015, methodology): “The idea is to compute the steepest descent direction for the weight update on the manifold deﬁned by these symmetries and use this direction to update the weights [4, 5, 6, 7].”
Data-Dependent Path Normalization in Neural Networks (2015, background): “Like the DDP, the Natural Gradients direction is a steepest descent direction, but it is based on a divergence measure calculated directly on the function fw, and not the parameterization w, and as such is invariant to reparametrizations.”
Exact natural gradient in deep linear networks and its application to the nonlinear case (2018, background): “However, besides some numerical evidence, the exact convergence rate of natural gradient remains unknown, and its implementation remains prohibitive due to its very expensive numerical computation [Pascanu and Bengio, 2013; Martens, 2014; Ollivier, 2015].”
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks (2019, background): “Natural gradient descent has an appealing interpretation as optimizing over a Riemannian manifold using an intrinsic distance metric; this implies the updates are invariant to transformations such as whitening [Ollivier, 2015, Luk and Grosse, 2018].”
Training recurrent networks online without backtracking (2015, unknown): “st steps. In practice we have used γt = O(1/ √ t) and Λ = (dimh).Id. The simplest and fastest way to approximate the Fisher matrix in Algorithm 2 is the outer product approximation (see discussion in [Oll15a]), which we have used in the experiments below. Namely, we simply use It ← ∂ℓt ∂yˆt ⊤∂ℓ t ∂y”
Degeneration in VAE: in the Light of Fisher Information Loss (2018, methodology): “Fisher Information can be applied to measure the quality of parameters in neural networks as mentioned in Desjardins et al. (2015); Ollivier (2015).”
Sigsoftmax: Reanalysis of the Softmax Bottleneck (2018, background): “Previous studies proposed alternative functions to softmax [8, 25, 27].”
Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis (2018, methodology): “These methods range from Ollivier (2015) who introduces a rank 1 update that accounts for the cross correlations between the biases and the weight matrices, to quasi Newton methods (Liu & Nocedal, 1989) that build a running estimate of the exact non-diagonal preconditioning matrix, and also include…”
Limitations of the Empirical Fisher Approximation (2019, methodology): “The generalized Gauss-Newton [Schraudolph, 2002] and natural gradient descent [Amari, 1998] methods have inspired a line of work on approximate second-order optimization [Martens, 2010, Botev et al., 2017, Park et al., 2000, Pascanu and Bengio, 2014, Ollivier, 2015].”