Asynchronous Stochastic Quasi-Newton MCMC for Non-Convex Optimization
By Umut Simsekli, Çağatay Yıldız, Thanh Huy Nguyen, Gaël Richard, Ali Taylan Cemgil. 2018
22 citations recorded by the source corpus.
Recent studies have illustrated that stochastic gradient Markov Chain Monte Carlo techniques have a strong potential in non-convex optimization, where local and global convergence guarantees can be shown under certain conditions. By building up on this recent theory, in this study, we develop an asynchronous-parallel stochastic L-BFGS algorithm for non-convex optimization. The proposed algorithm is suitable for both distributed and shared-memory settings. We provide formal theoretical analysis and show that the proposed method achieves an ergodic convergence rate of ${\cal O}(1/\sqrt{N})$ ($N$ being the total number of iterations) and it can achieve a linear speedup under certain conditions. We perform several experiments on both synthetic and real datasets. The results support our theory and show that the proposed algorithm provides a significant speedup over the recently proposed synchronous distributed L-BFGS algorithm.
Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Statistics and Probability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Kumar Kshitij Patel
- Mert Gurbuzbalaban
- Alireza Fallah
- Aymeric Dieuleveut
- Lingjiong Zhu
- Asuman E. Ozdaglar
- Umut Simsekli
- Sebastian U. Stich
In Statistics and Probability
- Kumar Kshitij Patel
- Asuman E. Ozdaglar
- Mert Gurbuzbalaban
- Umut Simsekli
- Lingjiong Zhu
- Aymeric Dieuleveut
- Francis R. Bach
Citations in the corpus
- Communication trade-offs for synchronized distributed SGD with large step size (2019, methodology): “Among second order methods, quasi newton methods like distributed L-BFGS Najafabadi et al. (2017); Şimşekli et al. (2018) are also widely popular among the machine learning community.”
- Robust Distributed Accelerated Stochastic Gradient Methods for Multi-Agent Networks (2019, background): “We observe that, the convergence behavior improves when we increase N from 4 to 5; however, further increasing N results in a degraded performance, since the overall computation time is dominated by the communication cost, a typical situation observed in synchronized distributed optimization (Kaya e”