Optimization for machine learning

By Suvrit Sra, Sebastian Nowozin, Stephen J. Wright. 2010

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

On Security and Sparsity of Linear Classifiers for Adversarial Settings (2016, background): “In particular, Xu et al. [13] have considered the following robust optimization problem: where ( z ) , and the so-called uncertainty set U is deﬁned as Typical examples of uncertainty sets according to the above deﬁnition include (cid:96) 1 and (cid:96) 2 balls [13,14].”
OptNet: Differentiable Optimization as a Layer in Neural Networks (2017, methodology): “…our scope (Boyd & Van-denberghe, 2004) but these methods have bound applicability in control frameworks (Morari & Lee, 1999; Sastry & Bodson, 2011); numerous statistical and mathematical formalisms (Sra et al., 2012), and physical simulation problems like rigid body dynamics (L ¨ otstedt, 1984).”
Automatic differentiation in machine learning: a survey (2015, background): “tion of gradients and Hessians of an objective function (Sra et al., 2011).”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, methodology): “For XAI, see [1, 2, 31, 55, 61, 67, 75, 111, 120, 128], for causal and counterfactual inference, see [63, 76, 131, 145, 146, 159, 185], and for optimization, see [32, 137, 166].”
Robust and Efficient Subspace Segmentation via Least Squares Regression (2012, background): “Theorem 1.9 in [26] shows that problem (17) is equivalent to a robust optimization problem, when the data X is subjected to a bounded matrix disturbance with a Frobenius norm.”
A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data: With applications in machine learning and signal processing (2015, unknown): “[3] S.”
A Unified Convergence Analysis for Shuffling-Type Gradient Methods (2020, methodology): “[40] S.”
On stochastic proximal gradient algorithms (2014, background): “Given θ n , compute θ n+1 = Prox γ n+1 (θ n + γ n+1 ∇(θ n )) .”
Deep learning for short-term traffic flow prediction (2016, methodology): “. Most of the recent improvements in large scale data analysis techniques come from applying scalable modern optimization and parallelization techniques, see Dean et al. (2012); Polson et al. (2015); Sra et al. (2012). Regularized Bayesian model ﬁtting allows to develop a parsimonious scalable model”
Optimization Methods for Large-Scale Machine Learning (2016, background): “There exist other sparsity-inducing regularizers besides the `1-norm, including group-sparsityinducing regularizers that combine `1- and `2-norms taken with respect to groups of variables [141, 7], as well as the nuclear norm for optimization over spaces of matrices [30].”
Proximal Methods for Hierarchical Sparse Coding (2010, methodology): “We consider two wav elet orthonormal bases, Haar and Daubechies3 (see Mallat, 1999), and choose a classical quad -tree structure on the coefficients, which has notably proven to be useful for image compression proble ms (Baraniuk, 1999). This experiment follows the approach of Zhao et al. (2009) who”
Regularizing deep networks using efficient layerwise adversarial training (2017, background): “Several regularization problems in machine learning such as ridge regression, lasso or robust SVMs have been shown to be instances of a more general robust optimization framework [19].”
Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection (2018, methodology): “Given the current A and Delta , and a set learning rate ∈ (0 , 1] , we can incrementally optimize W and b by performing the Gradient Descent (GD) algorithm [26].”
Playing with Duality: An overview of recent primal?dual approaches for solving large-scale optimization problems (2014, unknown): “Similarly, in fields like machine learning [3], [4], due to th e great ease with which data can now be collected and stored, quite often one has to cope with truly massive dat asets and to train very large models, which thus naturally lead to optimization problems of very high dimens io ality [5].”
Flexible Clustered Multi-Task Learning by Learning Representative Tasks (2016, background): “where @ Zi k k2 is the subgradient of Zi k k2 with respect to Zi and it is defined as [46]”
A simple algorithm for a class of nonsmooth convex-concave saddle-point problems (2015, background): “This model is very rich and encompasses most convex optimization models arising in a wide array of applications in signal/image processing and machine learning, see for instance the two very recent edited volumes [16, 22] and references therein.”
Towards Integrative Machine Learning and Knowledge Extraction (2017, background): “A different direction is thus to build on ideas from transfer learning [63].”