Strategies and Principles of Distributed Machine Learning on Big Data
By Eric P. Xing, Qirong Ho, Pengtao Xie, Wei Dai. 2015
167 citations recorded by the source corpus.
The rise of Big Data has led to new demands for Machine Learning (ML) systems to learn complex models with millions to billions of parameters, that promise adequate capacity to digest massive datasets and offer powerful predictive analytics thereupon. In order to run ML algorithms at such scales, on a distributed cluster with 10s to 1000s of machines, it is often the case that significant engineering efforts are required --- and one might fairly ask if such engineering truly falls within the domain of ML research or not. Taking the view that Big ML systems can benefit greatly from ML-rooted statistical and algorithmic insights --- and that ML researchers should therefore not shy away from such systems design --- we discuss a series of principles and strategies distilled from our recent efforts on industrial-scale ML solutions. These principles and strategies span a continuum from application, to engineering, and to theoretical research and development of Big ML systems and architectures, with the goal of understanding how to make them efficient, generally-applicable, and supported with convergence and scaling guarantees. They concern four key questions which traditionally receive little attention in ML research: How to distribute an ML program over a cluster? How to bridge ML computation with inter-machine communication? How to perform such communication? What should be communicated between machines? By exposing underlying statistical and algorithmic characteristics unique to ML programs but not typically seen in traditional computer programs, and by dissecting successful cases to reveal how we have harnessed these principles to design and develop both high-performance distributed ML software as well as general-purpose ML frameworks, we present opportunities for ML researchers and practitioners to further shape and grow the area that lies between ML and systems.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Cong Xie
- Oluwasanmi Koyejo
- Indranil Gupta
- Shengping Zhou
- Kevin Hsieh
- Onur Mutlu
- Jeroen Kloppenburg
- Matthijs Wolting
Citations in the corpus
- Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation (2019, background): “Further, even non-Byzantine-tolerant machine learning algorithms can naturally tolerate some noise in the input and during execution (Xing et al., 2016).”
- Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds (2017, background): “These works observe that in these ML algorithms, not all model parameters converge to their optimal value within the same number iterations — a property called non-uniform convergence [78].”
- Large-scale distributed L-BFGS (2017, background): “With the recent interest in Big Data analytics, it is critical to be able to scale machine learning techniques to train large-scale models [3].”
- A Survey on Distributed Machine Learning (2019, methodology): “Several communication management strategies [162] are used to spread and reduce the amount of data exchanged between machines:”
- FDML: A Collaborative Machine Learning Framework for Distributed Features (2019, background): “Depending on how the computation workload is partitioned [30], distributed machine learning systems can be categorized into data parallel and model parallel systems.”