Unwrapping ADMM: Efficient Distributed Computing via Transpose Reduction

By Tom Goldstein, Gavin Taylor, Kawika Barabin, Kent Sayre. 2015

19 citations recorded by the source corpus.

Recent approaches to distributed model fitting rely heavily on consensus ADMM, where each node solves small sub-problems using only local data. We propose iterative methods that solve {\em global} sub-problems over an entire distributed dataset. This is possible using transpose reduction strategies that allow a single node to solve least-squares over massive datasets without putting all the data in one place. This results in simple iterative methods that avoid the expensive inner loops required for consensus methods. To demonstrate the efficiency of this approach, we fit linear classifiers and sparse linear models to datasets over 5 Tb in size using a distributed implementation with over 7000 cores in far less time than previous approaches.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hanlin Tang
2. Ji Liu
3. Ce Zhang
4. Martin Jaggi
5. Dan Alistarh
6. Sebastian U. Stich
7. Zheng Xu
8. Tom Goldstein

In Computational Mechanics

1. Tom Goldstein
2. Hanlin Tang
3. Gavin Taylor
4. Mário A. T. Figueiredo
5. Ce Zhang
6. Ji Liu
7. Wotao Yin
8. Zheng Xu

Citations in the corpus

• Distributed Learning over Unreliable Networks (2018, methodology): “We adjust the learning rate by applying a linear scaling rule Goyal et al. (2017) and decay of 10 percent after 80 and 120 epochs, respectively. To achieve the best possible convergence, we apply a gradual warmup strategy Goyal et al. (2017) during the first 5 epochs.”
• Adaptive Consensus ADMM for Distributed Optimization (2017, background): “…involving a composite objective f ( v ) = i f i ( v ) , where worker i stores the data needed to compute f i , and so is well suited for distributed model fitting problems (Boyd et al., 2011; Zhang & Kwok, 2014; Song et al., 2016; Chang et al., 2016; Goldstein et al., 2016; Taylor et al., 2016).”
• Training Neural Networks Without Gradients: A Scalable ADMM Approach (2016, methodology): “This can be done effectively using transpose reduction strategies (Goldstein et al., 2016) that reduce the dimensionality of matrices before they are transmitted to a central node.”