Introduction to Optimization

By R. Venkata Rao. 2016

532 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Numerical Analysis

View Paper: https://www.semanticscholar.org/paper/b765caeaa0e4e134c0a9269ea380aba98b426bc4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Numerical Analysis

Citations in the corpus

Distributed stochastic optimization for deep learning (thesis) (2016, unknown): “In case that δh is chosen independently of h, the update is no different to the heavy ball method [44].”
An inertial forward–backward splitting method for solving inclusion problems in Hilbert spaces (2018, unknown): “In [2], Alvarez and Attouch employed the heavy ball method which was studied in [35,36] for maximal monotone operators by the proximal point algorithm.”
On the influence of momentum acceleration on online learning (2016, unknown): “If the probability distribution of the data, θ, is known beforehand, then one can evaluate J(w) and seek its minimizer by means of a variety of gradient-descent or Newton-type methods (Polyak, 1987; Bertsekas, 1999; Nesterov, 2004).”
Excess-Risk of Distributed Stochastic Learners (2013, unknown): “It is further shown in [31] and [36] that no algorithm can improve upon this rate under the same conditions.”
An introduction to continuous optimization for imaging (2016, unknown): “Ideally we want to choose α, β which solve min α,β f(xk + α∇f(xk) + β(xk − xk−1)) (see Polyak 1987).”
Adaptation, Learning, and Optimization over Networks (2014, unknown): “18) in the appendix that the cost function J(w) is said to be ν−strongly convex if, and only if, its Hessian matrix is sufficiently bounded away from zero [29, 45, 178, 191]: J(w) is ν−strongly convex ⇐⇒ ∇(2)w J(w) ≥ νIM > 0 (2.”
Relatively Smooth Convex Optimization by First-Order Methods, and Applications (2016, unknown): “then one can prove linear convergence for the gradient descent scheme (see [15], [19]), i.”
Fast optimization via inertial dynamics with closed-loop damping (2020, unknown): “Polyak [77, 78] and further studied by Attouch-Goudou-Redont [26] (exploration of local minima), Alvarez [7] (convergence in the convex case), Haraux-Jendoubi [64, 65] (convergence in the analytic case), Bégout-Bolte-Jendoubi [41] (convergence based on the Kurdyka-Lojasiewicz property), to cite part”
Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems (2013, methodology): “This latter algorithm is known as the heavy-ball method and significantly outperforms the standard gradient method on ill-conditioned problems [28].”
Rayleigh Quotient Based Optimization Methods For Eigenvalue Problems (2014, unknown): “However, it is suggested [23] that the local optimal CG (LOCG) method [39, 49] is more suitable for the symmetric eigenvalue problem.”
On Lower and Upper Bounds in Smooth and Strongly Convex Optimization (2016, unknown): “…iterative procedures and analyzing their convergence behavior accordingly, which dates back to the pioneering work of the Russian mathematician Lyapunov , has been successfully applied in the field of mathematical optimization many times, e.g., Polyak (1987) and more recently Lessard et al. (2014).”
ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs (2019, unknown): “For any δ ∈ (0 , C 1 ) , there is a large enough K ≥ 0 such that for all k ≥ K , With a standard recursion argument (e.g. Lemma 3 of [52]) we conclude max { 0 , V k − ν } → 0 .”
Adaptive Networks (2014, unknown): “Moreover, the convergence of stochastic-gradient algorithms will occur at the slower rate of O ð 1 = i Þ [26], [27].”