Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results

By Coralia Cartis, N. Gould, Philippe L. Toint. Mathematical programming, 2011

458 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Mechanics, Numerical Analysis

View Paper: https://www.semanticscholar.org/paper/e9a06e8e72bdb820eb60a72bc7ef3ef7abca18c0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Numerical Analysis

Citations in the corpus

Accelerated Methods for Non-Convex Optimization (2016, background): “ible to consider methods in which such subproblems are solved only approximately by a low complexity Hessian-free procedure. A number of researchers investigate this approach, including Cartis et al. [12] and Bianconcini et al. [7]. These works exhibit strong empirical results, but their analyses do”
A Distributed Second-Order Algorithm You Can Trust (2018, methodology): “Here we suggest updating σ at each iteration of the algorithm using an update rule inspired by trust-region methods (Cartis et al., 2011), where σ acts as the reciprocal of the trust-region radius.”
trlib: a vector-free implementation of the GLTR method for iterative solution of the trust region problem (2016, background): “Recently, trust region-like algorithms with guaranteed complexity estimates in relation to the KKT tolerance have been proposed [5, 6, 10].”
Recent advances in trust region algorithms (2015, methodology): “This special case was ﬁrst due to Griewank [66] and was considered later by many authors, see Nesterov and Polyak [81], Weiser et al. [134], and Cartis et al. [14].”
First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Time (2017, background): “he second-order information through Hessian or Hessianvector products (Nesterov and Polyak, 2006; Agarwal et al., 2017; Carmon et al., 2016; 2 Extracting Negative CurvatureFrom Noise Xu et al., 2017; Cartis et al., 2011a,b; Royer and Wright, 2017; Liu and Yang, 2017a,b), though many of them focused ”
Ellipsoidal Trust Region Methods and the Marginal Value of Hessian Information for Neural Network Training (2019, methodology): “Among the class of regularized Newton methods, trust region [22] and cubic regularization algorithms [16] are the most principled approaches, as they yield the strongest convergence guarantees.”
Adaptive norms for deep learning with regularized Newton methods (2019, methodology): “The most principled class of regularized Newton methods are trust region (TR) and adaptive cubic regularization algorithms (ARC) (Conn et al., 2000; Cartis et al., 2011), which repeatedly optimize a local Taylor model of the objective while making sure that the step does not travel too far such…”
Global linear convergence of Newton's method without strong-convexity or Lipschitz gradients (2018, background): “Otherwise, global convergence results of secondorder algorithms were known when f(x) has both strong-convexity and Lipschitz gradients ([22, 28]), or by solving cubic subproblems ([9, 24, 27]).”
Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity (2011, methodology): “In this paper and its predecessor [1], we explore a third alternative: employing a local cubic overestimator of the objective as a regularisation strategy for the step computation.”
Local Saddle Point Optimization: A Curvature Exploitation Approach (2018, methodology): “Among these approaches are trust-region methods that guarantee convergence to a second-order stationary point [8, 33, 6].”
On the Oracle Complexity of First-Order and Derivative-Free Algorithms for Smooth Nonconvex Minimization (2012, methodology): “As noted in Cartis et al. (2010a), conditions (2.”
PyHessian: Neural Networks Through the Lens of the Hessian (2019, background): “Hessian-based methods have also been explored for nonconvex problems, including trust-region (TR) [13], cubic regularization (CR) [36], and its adaptive variant (ARC) [11, 12].”