Exact natural gradient in deep linear networks and its application to the nonlinear case
By Alberto Bernacchia, Máté Lengyel, Guillaume Hennequin. 2018
75 citations recorded by the source corpus.
Subfields: Optimization & Training, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Optimization & Training
- Guodong Zhang
- Roger Baker Grosse
- Chaoqi Wang
- James Martens
- Guillaume Hennequin
- Alberto Bernacchia
- Lei Huang
- Gido M. van de Ven
In Scaling & Learning Theory
- James Martens
- Roger Baker Grosse
- Guodong Zhang
- Mikhail Khodak
- Chaoqi Wang
- Lester W. Mackey
- Alberto Bernacchia
- Nicoló Fusi
Citations in the corpus
- Initialization and Regularization of Factorized Neural Layers (2021, background): “Apart from Bernacchia et al. (2018) these largely examine existing algorithms, although Arora et al. (2018) do effectively pro-pose overcomplete knowledge distillation.”
- Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “In this work, we use a Kronecker-factored approximation (Martens and Grosse, 2015; Ritter et al., 2018).”
- Three Mechanisms of Weight Decay Regularization (2018, methodology): “(3)For K-FAC Gauss-Newton norm, we need the network to be linear due to the fact that K-FAC approximation is exact only for deep linear networks (Bernacchia et al., 2018).”
- Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks (2019, background): “In that case, we take the generalized inverse [Bernacchia et al., 2018] F † = n J (cid:62) G − 1 G − 1 J with G = JJ (cid:62) , which gives the following update rule: where u = [ u 1 , ..., u n ] (cid:62) = [ f ( θ , x 1 ) , ..., f ( θ , x n )] (cid:62) and y = [ y 1 , ..., y n ] (cid:62) .”
- Eigenvalue Corrected Noisy Natural Gradient (2018, unknown): “Bernacchia et al. [2018] showed that the Kronecker approximation is exact for deep linear networks, justifying the validity of the above assumption.”
- Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “One successful example is approximating the FIM of DNNs using the Kronecker product (K-FAC) [38], [39], [40], [41].”