Second-order Optimization for Neural Networks

By James Martens. 2016

67 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, background): “Since it is well-known that the curvature associated with neural network objective functions is highly non-diagonal ( e.g., Martens, 2016), a further requirement is that the matrix have off-diagonal terms.”
Continual Learning in Generative Adversarial Nets (2017, background): “Note that the above actually corresponds to an approximation known as the empirical Fisher information, which is more efﬁcient to compute [10].”
New Insights and Perspectives on the Natural Gradient Method (2014, background): “Martens (2016) reviews these approaches specifically in the context of neural networks.”
Improved multitask learning through synaptic intelligence (2017, methodology): “However, in contrast to EWC, here we are putting forward a method which allows for online computation of the importance measure, whereas EWC relies on the diagonal of the Fisher information metric at the ﬁnal parameters, which has to be computed during a separate phase at the end of each task.”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, methodology): “For any arbitrary loss function `, we have the so-called Gauss-Newton decomposition (Martens, 2016; Bottou et al., 2018) of the per-example Hessian∇(2)fi(θ): ∇(2)fi(θ) = `(zi; yi)∇θh(xi; θ)∇θh(xi; θ) + `(zi; yi)∇(2)θh(xi; θ) where zi = h(xi; θ) where ∇θh(xi; θ) ∈ R is the gradient of the network out”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, background): “The loss surface of deep neural networks has been widely reported to be ill-conditioned (LeCun et al., 2012; Martens, 2016).”
A Survey of Optimization Methods From a Machine Learning Perspective (2019, methodology): “Although computing and storing the Hessian matrices are difficult, with the development of research, the calculation of the Hessian matrix has made great progress [8], [9], [147], and the second-order optimization method has become more and more attractive.”
Revisiting Recursive Least Squares for Training Deep Neural Networks (2021, methodology): “Second-order algorithms typically use the second-order derivatives of the loss function to speed up the training of DNNs [14].”
Neural networks with late-phase weights (2020, background): “Despite its apparent simplicity, it remains a challenging optimization problem that has important similarities with stochastic minibatch training in deep neural networks (Schaul et al., 2013; Martens, 2016; Wu et al., 2018; Zhang et al., 2019a;b).”