Author manuscript, published in "International Conference on Machine Learning (ICML 2013) (2013)" Optimization with First-Order Surrogate Functions

2013

Citations in the corpus

• Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “In order to remain tractable in memory and computational overhead, optimization is performed in an adaptive low dimensional subspace determined by the history of gradients and positions. like IAG (Blatt et al., 2007), SAG (Roux et al., 2012), and MISO (Mairal, 2013; 2014) instead take update steps i”
• Minimizing finite sums with the stochastic average gradient (2013, background): “Indeed, there now exist proximal-gradient variants of SDCA [Shalev-Schwartz and Zhang, 2013a], MISO [Mairal, 2013], SVRG [Xiao and Zhang, 2014], and SAGA [Defazio et al.”
• Robust Aggregation for Federated Learning (2019, background): “In particular, it is the iterative minimization of a first-order surrogate in the sense of [63, 64].”
• Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learning (2014, methodology): “Their proofs can be found in the appendix of [34].”
• Finito: A faster, permutable incremental gradient method for big data problems (2014, methodology): “Our method can be seen as MISO, but with a step size scheme that gives neither a lower nor upper bound minimisation method.”
• Majorization-Minimization Algorithms in Signal Processing, Communications, and Machine Learning (2017, background): “For a comprehensive analysis of the above-described extensions, we refer the reader to [69], [70], [72], and [73].”
• Parallel Successive Convex Approximation for Nonsmooth Nonconvex Optimization (2014, background): “This assumption is satisfied in many practical problems suchas (group) Lasso.”
• Stochastic Recursive Gradient Algorithm for Nonconvex Optimization (2017, background): “Recently, a large number of improved variants of stochastic gradient algorithms have emerged, including SAG/SAGA [22, 4], MISO/FINITO [13, 5], SDCA [24], SVRG/S2GD [9, 10], SARAH [16] 1.”
• StopWasting My Gradients: Practical SVRG (2015, methodology): “Other methods have subsequently been shown to have this property [2, 3, 4], but these all require storing a previous evaluation of the gradient f ′ i or the dual variables for each i.”
• On the Acceleration of L-BFGS with Second-Order Information and Stochastic Batches (2018, unknown): “Numerous work has focused on the choice of g k such as SAG/SAGA [30, 7], MISO/FINITO [17, 8], SDCA [32], SVRG/S2GD [11, 13], SARAH [23, 24].”
• Fast Asynchronous Parallel Stochastic Gradient Descent: A Lock-Free Approach with Convergence Guarantee (2016, background): “…stochastic gradient descent (SGD) and its variants (Xiao 2009; Duchi and Singer 2009; Roux, Schmidt, and Bach 2012; Johnson and Zhang 2013; Mairal 2013; Shalev-Shwartz and Zhang 2013; Liu et al. 2014; Nitanda 2014; Zhang and Kwok 2014) have recently attracted much attention to solve…”
• Iterative Splits of Quadratic Bounds for Scalable Binary Tensor Factorization (2014, methodology): “This algorithm is sometimes referred as Majorization-Minimization algorithm in the literature [16].”
• Federated Multi-Task Learning under a Mixture of Distributions (2021, background): “Our algorithms can easily be adapted to solve more general problems in a novel framework, which can be seen as a federated extension of the centralized surrogate optimization approach in [43].”
• A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data: With applications in machine learning and signal processing (2015, methodology): “Practically, it has been used in emerging large-scale signal processing and machine learning applications, such as compressive sensing/sparse signal recovery [23], [24], matrix completion [25], [26], tensor decomposition [27], [28], to name just a few.”
• One Method to Rule Them All: Variance Reduction for Data, Parameters and Many New Methods (2019, background): “However, a remedy does exist: the situation with non-convergence can be resolved by using one of the many variancereduction strategies for constructing g developed over the last several years [34, 3, 12, 19, 35].”