Exploiting iterative-ness for parallel ML computations

By Henggang Cui, Alexey Tumanov, Jinliang Wei, Lianghong Xu, Wei Dai, Jesse Haber-Kucharsky, Qirong Ho, G. R. Ganger, Phillip B. Gibbons, Garth A. Gibson and 1 more. 2014

40 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Hardware and Architecture, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. H. Zhang
2. Eric P. Xing
3. Wei Dai
4. Phillip B. Gibbons
5. Roy H. Campbell
6. Kevin Hsieh
7. Onur Mutlu
8. G. R. Ganger

In Hardware and Architecture

1. Phillip B. Gibbons
2. Onur Mutlu
3. Wei Dai
4. Eric P. Xing
5. G. R. Ganger
6. Jinliang Wei
7. Henggang Cui
8. Garth A. Gibson

In Information Systems

1. Wei Dai
2. Roy H. Campbell
3. Eric P. Xing
4. Onur Mutlu
5. G. R. Ganger
6. Jinliang Wei
7. Henggang Cui
8. Phillip B. Gibbons

Citations in the corpus

• The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “…that aim to enable efficient ML training with centralized datasets (e.g., (Recht et al., 2011; Low et al., 2012; Chilimbi et al., 2014; Dean et al., 2012; Ho et al., 2013; Cui et al., 2014; Li et al., 2014a; Chen et al., 2015; Goyal et al., 2017)), where the data re-side within a single data center.”
• GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server (2016, methodology): “Fortunately, iterative applications like neural network training typically apply the same parameter data accesses every iteration [11], so the parameter server can easily predict the Read operations and perform them in advance in the background.”
• Addressing the straggler problem for iterative convergent parallel ML (2016, methodology): “Instead, these frameworks share state and exploit ML-specific properties to reduce coordination overheads and converge far faster [1, 3, 11, 12, 15, 35, 38].”
• TicTac: Accelerating Distributed Deep Learning with Communication Scheduling (2018, methodology): “Smarter interleaving of computation and communication : This approach is adopted by several layer-by-layer systems (Arnold, 2016; Cui et al., 2016; Zhang et al., 2017) where the models are sequential and obtaining the order is trivial (Cui et al., 2014).”
• Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds (2017, background): “We then discuss WAN bandwidth constraints and study the performance implications of running two state-of-the-art ML systems over WANs.”