Proximal Algorithms in Statistics and Machine Learning
By Nicholas G. Polson, James G. Scott, Brandon T. Willard. 2015
169 citations recorded by the source corpus.
In this paper we develop proximal methods for statistical learning. Proximal point algorithms are useful in statistics and machine learning for obtaining optimization solutions for composite functions. Our approach exploits closed-form solutions of proximal operators and envelope representations based on the Moreau, Forward-Backward, Douglas-Rachford and Half-Quadratic envelopes. Envelope representations lead to novel proximal algorithms for statistical optimisation of composite objective functions which include both non-smooth and non-convex objectives. We illustrate our methodology with regularized Logistic and Poisson regression and non-convex bridge penalties with a fused lasso norm. We provide a discussion of convergence of non-descent algorithms with acceleration and for non-convex functions. Finally, we provide directions for future research.
Subfields: Applied Mathematics, Computational Mechanics, Computational Theory and Mathematics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Applied Mathematics
- Bryan T. Kelly
- Jan Hendrik Witte
- J. B. Heaton
- Shihao Gu
- D. Xiu
- Laurent Condat
- Pascal Fua
- Akira Hirabayashi
In Computational Mechanics
- Peter Richtárik
- Konstantin Mishchenko
- Grigory Malinovsky
- Adil Salim
- Laurent Condat
- Akira Hirabayashi
- Daichi Kitahara
- Andres A Contreras Marcillo
In Computational Theory and Mathematics
- Peter Richtárik
- Konstantin Mishchenko
- Adil Salim
- Grigory Malinovsky
- Akira Hirabayashi
- Daichi Kitahara
- Andres A Contreras Marcillo
- Bryan T. Kelly
Citations in the corpus
- Deep Learning in Finance (2016, methodology): “Traversing the objective function is the desired problem, and handling the multi-modal and slow convergence of traditional decent methods can be alleviated with proximal algorithms such as the alternating method of multipliers (ADMM), as has been discussed in Polson et al. (2015 a, b).”
- Proximal Splitting Algorithms: A Tour of Recent Advances, with New Twists. (2020, background): “Many problems in statistics, machine learning, signal and image processing, or control can be formulated as convex optimization problems [1–7].”
- Distributed Proximal Splitting Algorithms with Rates and Acceleration (2020, background): “2 Related Work Many estimation problems in a wide range of scientific fields can be formulated as large-scale convex optimization problems (Palomar and Eldar, 2009; Sra et al., 2011; Bach et al., 2012; Bubeck, 2015; Polson et al., 2015; Chambolle and Pock, 2016a; Glowinski et al., 2016; Stathopoulos”
- Empirical Asset Pricing via Machine Learning (2018, methodology): “B.1 Lasso, Ridge, Elastic Net, and Group Lasso We present the accelerated proximal algorithm (APG), see, e.g., Parikh and Boyd (2013) and Polson et al. (2015)., which allows for efficient implementation of the elastic net, Lasso, Ridge regression, and Group Lasso for both l2 and Huber losses.”
- Dualize, Split, Randomize: Fast Nonsmooth Optimization Algorithms (2020, background): “Many problems in statistics, machine learning, or signal processing can be formulated as highdimensional convex optimization problems [3,12,56,59,68,69].”
- Kullback-Leibler Proximal Variational Inference (2015, methodology): “See [16] for a good review of proximal methods for machine learning.”