Scaling Up Quasi-newton Algorithms: Communication Efficient Distributed SR1

By Majid Jahani, M. Nazari, Sergey G. Rusakov, Albert S. Berahas, Martin Tak'avc. 2019

17 citations recorded by the source corpus.

In this paper, we present a scalable distributed implementation of the sampled LSR1 (S-LSR1) algorithm. First, we show that a naive distributed implementation of S-LSR1 requires multiple rounds of expensive communications at every iteration and thus is inefficient. We then propose DS-LSR1, a communication-efficient variant of the S-LSR1 method, that drastically reduces the amount of data communicated at every iteration, that has favorable work-load balancing across nodes and that is matrix-free and inverse-free. The proposed method scales well in terms of both the dimension of the problem and the number of data points. Finally, we illustrate the performance of DS-LSR1 on standard neural network training tasks.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

rTop-k: A Statistical Estimation Approach to Distributed SGD (2020, neutral): “SGD methods [1], [3]–[15], as well as other fundamental contributions to distributed first-order [16]–[20] and secondorder [21]–[25] optimization procedures.”
Efficient Distributed Hessian Free Algorithm for Large-scale Empirical Risk Minimization via Accumulating Sample Strategy (2018, background): “Recently, several papers by [Byrd et al., 2016, Berahas et al., , Jahani et al., 2019, Berahas et al., 2016, Schraudolph et al., 2007, Berahas et al., 2019, Mokhtari and Ribeiro, 2015, Roosta-Khorasani and Mahoney, 2018] have studied how to apply second orders methods to solve ERM a r X i v : 1810 .”