Petuum: A New Platform for Distributed Machine Learning on Big Data

By Eric P. Xing, Qirong Ho, Wei Dai, Jin Kyu Kim, Jinliang Wei, Seunghak Lee, Xun Zheng, Pengtao Xie, Abhimanu Kumar, Yaoliang Yu. 2013

526 citations recorded by the source corpus.

What is a systematic way to efficiently apply a wide spectrum of advanced ML programs to industrial scale problems, using Big Models (up to 100 s of billions of parameters) on Big Data (up to terabytes or petabytes)? Modern parallelization strategies employ fine-grained operations and scheduling beyond the classic bulk-synchronous processing paradigm popularized by MapReduce, or even specialized graph-based execution that relies on graph representations of ML programs. The variety of approaches tends to pull systems and algorithms design in different directions, and it remains difficult to find a universal platform applicable to a wide range of ML programs at scale. We propose a general-purpose framework, Petuum, that systematically addresses data- and model-parallel challenges in large-scale ML, by observing that many ML programs are fundamentally optimization-centric and admit error-tolerant, iterative-convergent algorithmic solutions. This presents unique opportunities for an integrative system design, such as bounded-error network synchronization and dynamic scheduling based on ML program structure. We demonstrate the efficacy of these system designs versus well-known implementations of modern ML algorithms, showing that Petuum allows ML programs to run in much less time and at considerably larger model sizes, even on modestly-sized compute clusters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

In ML Systems & Hardware

Citations in the corpus

Memory-Efficient Pipeline-Parallel DNN Training (2020, neutral): “With data parallelism [22], every worker has a copy of the entire model and the input dataset is sharded across workers.”
Exploiting Unintended Feature Leakage in Collaborative Learning (2018, background): “Several architectures have been proposed for distributed, collaborative, and federated learning [9, 11, 33, 38, 62, 68]: with and without a central server, with different ways of aggregating models, etc.”
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (2017, background): “Large-scale distributed training improves the productivity of training deeper and larger models (Chilimbi et al., 2014; Xing et al., 2015; Moritz et al., 2015; Zinkevich et al., 2010).”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, background): “The training of large-scale models with huge amounts of data are often carried on distributed systems [1][2][3][4][5][6][7][8][9], where data parallelism is adopted to exploit the compute capability empowered by multiple workers [10].”
Gradient Sparsification for Communication-Efficient Distributed Optimization (2017, background): “Modern large-scale machine learning applications require scaling stochastic optimization algorithms [30, 28, 15, 11] to distributed computational architectures [10, 39, 12, 20, 19, 38] or multicore systems [26, 9, 22, 25].”
The Convergence of Sparsified Gradient Methods (2018, background): “One can classify proposed solutions into a) lossless, either based on factorization [29, 6] or on executing SGD with extremely large batches, e.”
Clipper: A Low-Latency Online Prediction Serving System (2016, background): “Parameter Server: There has been considerable work in the learning systems community on parameterservers [5, 21, 38, 62].”
Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback (2019, background): “To achieve state-of-the-art performance, they often have to leverage the computing power of multiple machines during training [8, 26, 28, 6].”
GLMix: Generalized Linear Mixed Models For Large-Scale Response Prediction (2016, background): “The use of a parameter server [23, 13] is an active research area, which can be used to ﬁt models with a large number of features.”
Multi-Agent Distributed Lifelong Learning for Collective Knowledge Acquisition (2017, methodology): “Similar to ADMM, we first need to form the augmented Lagrangian JT (L1, . . . ,LN ,Z ) for problem (6) at time t in order to replace the constrained problem by an unconstrained objective function which has an added penalty term: JT (L1, . . . ,LN ,Z ) = 1 T T∑ t=1 N∑ i=1 F (t ) i (Li )+ λ∥Li ∥2F + ⟨”
OD-SGD (2020, background): “Under this circumstance, distributed training becomes the prevailing practice and it greatly improves the productivity of training deeper and larger neural networks [7, 8, 9, 10].”
Learning Transferable Knowledge Through Embedding Spaces (2019, methodology): “Some approaches assume there is a central server (or a group of server nodes) in the network, and the worker agents transmit locally learned information to the server(s), which then perform knowledge fusion [245].”
Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2022, background): “Overall, there are three key approaches: Synchronous SGD [51], Stale asynchronous SGD [61], and Asynchronous SGD [11].”