High-Performance Distributed ML at Scale through Parameter Server Consistency Models

By Wei Dai, Abhimanu Kumar, Jinliang Wei, Qirong Ho, Garth A. Gibson, Eric P. Xing. 2014

123 citations recorded by the source corpus.

As Machine Learning (ML) applications embrace greater data size and model complexity, practitioners turn to distributed clusters to satisfy the increased computational and memory demands. Effective use of clusters for ML programs requires considerable expertise in writing distributed code, but existing highly-abstracted frameworks like Hadoop that pose low barriers to distributed-programming have not, in practice, matched the performance seen in highly specialized and advanced ML implementations. The recent Parameter Server (PS) paradigm is a middle ground between these extremes, allowing easy conversion of single-machine parallel ML programs into distributed ones, while maintaining high throughput through relaxed ``consistency models" that allow asynchronous (and, hence, inconsistent) parameter reads. However, due to insufficient theoretical study, it is not clear which of these consistency models can really ensure correct ML algorithm output; at the same time, there remain many theoretically-motivated but undiscovered opportunities to maximize computational throughput. Inspired by this challenge, we study both the theoretical guarantees and empirical behavior of iterative-convergent ML algorithms in existing PS consistency models. We then use the gleaned insights to improve a consistency model using an "eager" PS communication mechanism, and implement it as a new PS system that enables ML programs to reach their solution more quickly.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Federated Learning: Challenges, Methods, and Future Directions (2019, background): “usceptible to stragglers in the face of device variability. Asynchronous schemes are an attractive approach to mitigate stragglers in heterogeneous environments, particularly in shared-memory systems [27, 30, 48, 92, 141]. However, they typically rely on bounded-delay assumptions to control the degr”
Petuum: A New Platform for Distributed Machine Learning on Big Data (2013, methodology): “exploited to speed up distributed ML programs: dependency structures between parameters, non-uniform convergence of parameters, and a limited degree of error tolerance [13], [14], [17], [21], [23], [24].”
Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, background): “Notably, communication heterogeneity can be viewed as a strategy to address the differences in device computing power [38] by using bounded-delay assumptions to control devices latencies.”
Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters (2017, methodology): “Unless otherwise speciﬁed, our discussion in this paper assumes synchronous replication of model parameters in each training iteration, although we note that Poseidon’s design can easily be applied to asynchronous or bounded-asynchronous consistency models [12, 8].”
On Model Parallelization and Scheduling Strategies for Distributed Machine Learning (2014, methodology): “The recently emerged parameter server architecture [15, 18] has started to pave ways for a unified programming interface for data parallel algorithms, based on various parallellization models such as stale synchroneous parallelism (SSP) [15], eager SSP [5], and value-bound asynchronous parallelism [”
Toward Understanding the Impact of Staleness in Distributed Machine Learning (2018, background): “In practice the staleness experienced during run time can be limited to small staleness (Dai et al., 2015), which minimizes the likelihood of negative gradient coherence during the early phase.”
Distributed Machine Learning in Coalition Environments: Overview of Techniques (2018, background): “When the value of τ is bounded, it can be shown that under some convexity assumptions, distributed gradient descent converges to the optimal loss function value as the number of iterations goes to inﬁnity [2], [11]–[14].”
Hop: Heterogeneity-aware Decentralized Training (2019, background): “Current available protocols mainly include Bulk Synchronous Parallel (BSP) [8,29,32], Stale Synchronous Parallel (SSP) [14,25,30] and asynchronous coordination [5,9,23].”
Strategies and Principles of Distributed Machine Learning on Big Data (2015, methodology): “Such early aggregation is preferred for centralized parameter storage paradigms that communicate full parameters A from servers to workers [37,40], and it is natural to ask if there are other strategies that may perhaps be better suited to different storage paradigms.”
Fault Tolerance in Iterative-Convergent Machine Learning (2018, background): “Reduced precision (Example 2.1) and bounded staleness consistency (Example 2.2) have already been the focus of much attention in both the ML and systems communities (Zhang et al., 2017a; Jia et al., 2018; Wei et al., 2015; Dai et al., 2015).”