Addressing the straggler problem for iterative convergent parallel ML

By Aaron Harlap, Henggang Cui, Wei Dai, Jinliang Wei, G. R. Ganger, Phillip B. Gibbons, Garth A. Gibson, Eric P. Xing. 2016

148 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Hardware and Architecture, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In Information Systems

Citations in the corpus

Fault Tolerance in Iterative-Convergent Machine Learning (2018, background): “…a cluster resource allocator (Vavilapalli et al., 2013; Hindman et al., 2011), parameter synchronization can be bottlenecked on a slow or congested network (Li et al., 2014b; Zhang et al., 2017b), and stragglers can severely impact overall job throughput (Cipar et al., 2013; Harlap et al., 2016).”
Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark (2018, background): “In recent years, researchers have proposed a range of hardware, software, and statistical optimizations for deep learning, ranging from new software systems [2, 8, 9, 13, 25] and hardware [7, 18, 26, 38] to communication methods [12, 20, 37, 52] and training algorithms [15, 17, 23, 28, 44–46].”
Heterogeneity-aware Distributed Parameter Servers (2017, methodology): “The SSP-based Petuum is 1.57 × faster than its ASP version, and FlexRR further brings a 20% improvement which is similar to their reported results [21].”
Rateless codes for near-perfect load balancing in distributed matrix-vector multiplication (2018, background): “Such work stealing or dynamic load balancing strategies are often implemented in shared and distributed memory settings [12, 13, 26].”
DAWNBench : An End-to-End Deep Learning Benchmark and Competition (2017, methodology): “Researchers have proposed new software systems [1, 7, 8, 11, 23, 39], training algorithms [12, 15, 21, 22, 26, 35–38, 40, 42], communication methods [8, 10, 11, 19, 32, 41] and hardware [6, 16–18, 24, 30] to decrease this cost.”
Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds (2017, methodology): “We use the following algorithm convergence criterion, based on guidance from our ML experts: if the value of the objective function (the objective value) in an algorithm changes by less than 2% over the course of 10 iterations, we declare that the algorithm has converged [32].”
TiFL: A Tier-based Federated Learning System (2020, background): “[13] propose a novel design named RapidReassignment to handle straggles by specializing work shedding.”