Exploiting Bounded Staleness to Speed Up Big Data Analytics

By Henggang Cui, James Cipar, Qirong Ho, Jin Kyu Kim, Seunghak Lee, Abhimanu Kumar, Jinliang Wei, Wei Dai, G. R. Ganger, Phillip B. Gibbons and 2 more. 2014

171 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms (2018, methodology): “Synchronization delays can be alleviated via asynchronous gradient aggregation (Recht et al., 2011; Cui et al., 2014; Gupta et al., 2016; Mitliagkas et al., 2016; Dutta et al., 2018).”
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD (2018, methodology): “…Federated Learning (McMahan et al., 2016), Elastic-Averaging (Zhang et al., 2015) and Decentralized averaging (Jiang et al., 2017; Lian et al., 2017), as well as synchronous/asynchronous distributed SGD with a central parameter server (Dean et al., 2012; Cui et al., 2014; Dutta et al., 2018).”
Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study (2015, background): “Orthogonal to the system design, many researchers have proposed solutions to counter staleness in the system, such as bounding the staleness in the system [6] or changing optimization objective function, such as elastic averaging SGD [25].”
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping (2020, background): “There are two CLS before crossover ( CLS 1 and CLS 2 ) and four different tensor sizes: 1MB, 2MB, 3MB and 4MB. CLS 1 = [ 1 , 1 , 1 , 1 ] means that all four tensor sizes belong to the same size-class, 4MB, and CNT 1 = [ 4 ] indicates that there are four 4MB tensor objects allocated.”
PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018, background): “4 PipeDream’s default semantics (weight stashing but no vertical sync) are between regular minibatched SGD on a single machine, and data parallelism with BSP synchronization [8, 18].”
Addressing the straggler problem for iterative convergent parallel ML (2016, methodology): “While FlexRR’s support for SSP is similar to recent systems [14, 35], its scalable, data-centric work reassignment mechanism is new.”
Fault Tolerance in Iterative-Convergent Machine Learning (2018, methodology): “unreliability and resource limitation issues, such as bounded staleness consistency (Ho et al., 2013; Cipar et al., 2013; Cui et al., 2014), quantization and low-precision arithmetic (Courbariaux et al.”
High-Performance Distributed ML at Scale through Parameter Server Consistency Models (2014, background): “One of the primary challenges in data parallelism is designing software systems to share the ML model across a cluster of commodity machines, in a manner that is theoretically well-justified, empirically highperformance, and easy-to-program all at the same time — and a highly promising solution is t”
PipeDream: generalized pipeline parallelism for DNN training (2019, background): “PipeDream’s default semantics (weight stashing but no vertical sync) are between regular minibatched SGD on a single worker, and data parallelism with BSP synchronization [19, 27].”
Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems (2020, methodology): “Our distributed training framework falls in the data-parallel paradigm (Li et al., 2014; Cui et al., 2014; 2016; Luo et al., 2018).”
Scalable deep learning on distributed GPUs with a GPU-specialized parameter server (2015, methodology): “Consistent reports indicate that, in such an architecture, some degree of asynchrony (bounded or not) in parameter update exchanges among workers leads to signiﬁcantly faster convergence than when using BSP [17, 21, 7, 13, 10, 4, 27].”
Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds (2017, background): “…running across 11 Amazon EC2 global regions and on a cluster that emulates EC2 WAN bandwidth show that Gaia provides 1.8–53.5× speedup over two state-of-the-art distributed ML systems, and is within 0.94–1.40× of the speed of running the same ML algorithm on machines on a local area network (LAN).”
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism (2020, unknown): “analyze the effects of bounded staleness [9].”
GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server (2016, methodology): “Figure 14 compares the per-mini-batch AdamLike model training time with 8 machines, when using GeePS, GeePSsingle-table, and CPU-PS, and each of three synchronization models: BSP (“Slack 0”), SSP (“Slack 1”), and Asynchronous (“Slack Inf”).”
Asynchronous Byzantine Machine Learning (2018, background): “Stale-synchronous parallel (Ho et al., 2013; Cui et al., 2014) or bulk-synchronous (Zinke-vich et al., 2010; Chen et al., 2016) variants typically target settings with limited staleness due to the limited performance variability among the computing devices.”
Toward Understanding the Impact of Staleness in Distributed Machine Learning (2018, methodology): “Non-synchronous execution can improve system throughput due to lower synchronization overheads, which is well understood (Ho et al., 2013; Chen et al., 2016; Cui et al., 2014; Chilimbi et al., 2014).”