Numerical optimization
By Jorge Nocedal, Stephen J. Wright. 2004
17,888 citations recorded by the source corpus.
Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization. It responds to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems. For this new edition the book has been thoroughly updated throughout. There are new chapters on nonlinear interior methods and derivative-free methods for optimization, both of which are used widely in practice and the focus of much current research. Because of the emphasis on practical methods, as well as the extensive illustrations and exercises, the book is accessible to a wide audience. It can be used as a graduate text in engineering, operations research, mathematics, computer science, and business. It also serves as a handbook for researchers and practitioners in the field. The authors have strived to produce a text that is pleasant to read, informative, and rigorous - one that reveals both the beautiful nature of the discipline and its practical side.
Subfields: Atomic and Molecular Physics, and Optics, Computational Mechanics, Computational Theory and Mathematics, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Atomic and Molecular Physics, and Optics
- L. Vanderstraeten
- J. Haegeman
- Frank Verstraete
- Eldad Haber
- Michael J. Black
- Francis R. Bach
- L. Eon Bottou
- James Martens
In Computational Mechanics
- Peter Richtárik
- Nicolas Loizou
- Jakub Konecný
- Simon Lacoste-Julien
- Martin Takác
- Francis R. Bach
- Michael I. Jordan
- Martin Jaggi
In Computational Theory and Mathematics
- Peter Richtárik
- Nicolas Loizou
- Simon Lacoste-Julien
- Francis R. Bach
- Sergey Levine
- Jakub Konecný
- Brendan O'Donoghue
- Michael I. Jordan
In Optimization & Training
- Peter Richtárik
- Simon Lacoste-Julien
- Nicolas Loizou
- Jakub Konecný
- Francis R. Bach
- Martin Jaggi
- Martin Takác
- Sebastian U. Stich
Citations in the corpus
- Learning to learn by gradient descent by gradient descent (2016, methodology): “Such a memory, if appropriately designed could allow the optimizer to learn algorithms similar to (low-memory) approximations to Newton’s method, e.g. (L)BFGS [see Nocedal and Wright, 2006].”
- High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, methodology): “We compute an approximate solution to the trust region problem using the conjugate gradient algorithm (Wright & Nocedal, 1999).”
- Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “…has been substantial work in explicitly incorporating second-order information through the Hessian matrix in Newton methods (Liu and Nocedal, 1989; Nocedal and Wright, 2006; Martens, 2010) or through the Fisher information matrix in natural gradient meth-ods (Amari, 1998; Roux et al., 2008;…”
- Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “The proof is in Appendix A and follows the deterministic case [61].”
- Optimization Methods for Large-Scale Machine Learning (2016, methodology): “ampling is robust and eective [2,29,122,132]. An inexact Newton method that incorporates this techniques is outlined in Algorithm6.1. The algorithm is stated with a backtracking (Armijo) line search [114], though other stepsize-selection techniques could be considered as well. Algorithm 6.1 Subsampl”
- Minimizing finite sums with the stochastic average gradient (2013, methodology): “Stochastic versions of FG methods: Various options are available to accelerate the convergence of the FG method for smooth functions, such as the accelerated full gradient (AFG) method of Nesterov [1983], as well as classical techniques based on quadratic approximations such as diagonallyscaled FG m”
- Privacy-Preserving Distributed Linear Regression on High-Dimensional Data (2017, methodology): “The CGD algorithm can be interpreted as solving the system by iteratively minimizing the objective ‖Aθ − b‖2 with respect to the parameter vector θ using the method of conjugate gradients [60].”
- Machine Learning Models that Remember Too Much (2017, methodology): “we can relax it to an unconstrained optimization problem using the penalty function method [60].”
- Learning explanations that are hard to vary (2020, background): “designed to minimize the pooled loss as fast as possible4. Indeed, a step of GD is equivalent to finding a tight5 quadratic upper bound L^to L, and then jumping to the minimizer of this approximation [Nocedal and Wright, 2006]. While speed is often desirable, note that by construction GD completely i”
- New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “…include Tikhonov regularization/damping and the closely related trustregion method (e.g. Tikhonov, 1943; Moré and Sorensen, 1983; Conn et al., 2000; Nocedal and Wright, 2006), and other ones such as the “structural damping” approach of Martens and Sutskever (2011), or the approach present in…”
- Distributed optimization with arbitrary local solvers (2015, methodology): “CD Coordinate Descent [51] APPROX Accelerated, Parallel and Proximal Coordinate Descent [20, 21] GD Gradient Descent with Backtracking Line Search [44] CG Conjugate Gradient Method [25] L-BFGS Quasi-Newton with Limited-Memory BFGS Updating [10] BB Barzilai-Borwein Gradient Method [4] FISTA Fast Iter”
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (2017, methodology): “In the setting of least-squares function approximation, the second-order algorithm of choice is commonly Gauss-Newton, which approximates the curvature as the Gauss-Newton matrix G := E[JJ ], where J is the Jacobian of the mapping from parameters to outputs [19].”
- A Kronecker-factored approximate Fisher matrix for convolution layers (2016, background): “For the remainder of this paper, we will assume the network’s prediction f(x,θ) determines the value of the parameter z of a distribution Ry|z over y, and the loss function is the corresponding negative log-likelihood L(y, z) = − log r(y|z).”
- DAGs with NO TEARS: Continuous Optimization for Structure Learning (2018, methodology): “To handle the nonconvexity, a slight modification (Nocedal and Wright, 2006, Procedure 18.2) needs to be applied.”