A Survey on Distributed Machine Learning

By Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, Jan S. Rellermeyer. ACM Computing Surveys, 2019

881 citations recorded by the source corpus.

The demand for artificial intelligence has grown significantly over the past decade, and this growth has been fueled by advances in machine learning techniques and the ability to leverage hardware acceleration. However, to increase the quality of predictions and render machine learning solutions feasible for more complex applications, a substantial amount of training data is required. Although small machine learning models can be trained with modest amounts of data, the input for training larger models such as neural networks grows exponentially with the number of parameters. Since the demand for processing training data has outpaced the increase in computation power of computing machinery, there is a need for distributing the machine learning workload across multiple machines, and turning the centralized into a distributed system. These distributed systems present new challenges: first and foremost, the efficient parallelization of the training process and the creation of a coherent model. This article provides an extensive overview of the current state-of-the-art in the field by outlining the challenges and opportunities of distributed machine learning over conventional (centralized) machine learning, discussing the techniques used for distributed machine learning, and providing an overview of the systems that are available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f9a855ae59579d16dca6a5133cd8daddd3305582

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yong Xiang
2. Weihua Zhuang
3. Xuemin Shen
4. Jie Gao
5. Jiankun Hu
6. Dejing Dou
7. Haoyi Xiong
8. Wen Wu

In Computer Vision and Pattern Recognition

1. Yong Xiang
2. Weihua Zhuang
3. Xuemin Shen
4. Jie Gao
5. Dejing Dou
6. Haoyi Xiong
7. Wen Wu
8. Chenhao Xu

Citations in the corpus

• Holistic Network Virtualization and Pervasive Network Intelligence for 6G (2023, background): “Model Training - Due to the distributed data and user privacy concerns, distributed training is suitable for training AI models in a network [206].”
• From distributed machine learning to federated learning: a survey (2021, methodology): “Vertical FL utilizes model parallelism to distribute parallel paths of a single model tomultiple devices in order to handle the data of different features [170].”
• Asynchronous Federated Learning on Heterogeneous Devices: A Survey (2021, background): “Distributed ML (DML) is a research topic that investigates different structures or topologies of the computer cluster for better training machine learning models [22].”
• Edge Learning for B5G Networks With Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing (2022, neutral): “We summarize three types of topologies for developing distributed ML according to the degrees of being distributed as characterized in [19].”
• A Comprehensive Survey of Privacy-preserving Federated Learning (2021, background): “There are typically two distribution schemes: data parallel and model parallel [178].”