Demystifying Parallel and Distributed Deep Learning

By Tal Ben-Nun, Torsten Hoefler. 2018

809 citations recorded by the source corpus.

Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this survey, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. We present trends in DNN architectures and the resulting implications on parallelization strategies. We then review and model the different types of concurrency in DNNs: from the single operator, through parallelism in network inference and training, to distributed deep learning. We discuss asynchronous stochastic optimization, distributed system architectures, communication schemes, and neural architecture search. Based on those approaches, we extrapolate potential directions for parallelism in deep learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Optimization & Training

Citations in the corpus

ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep learning (2021, background): “We refer the reader to Ben-Nun and Hoefler [19] for a thorough survey on parallelism in DL.”
Gradient Descent with Compressed Iterates (2019, methodology): “[3, 19] and combinations of decentralization and gradient quantization have been studied in recent work, see e.”
Machine Unlearning (2019, background): “Instead, in typical distributed training algorithms [38], [39], training updates from each constituent are shared between each other and training iterates based on a joint update .”
Parallel convolutional processing using an integrated photonic tensor core (2021, background): “The increased demand for machine learning on very large datasets 2 and the growing offering of AI services on the cloud 8,9 has driven a resurgence in custom hardware designed to accelerate MAC com-putations—the fundamental mathematical element needed for matrix-vector multiplication (MVM)…”
Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020, background): “…Learning is a related ﬁeld with similarities and key differences: while both study the theory and practice involving partitioning of data and aggregation of model updates [19, 9, 37, 70], federated learning is additionally concerned with data that is private and distributed in a non-i.i.d. fashion.”
HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients (2020, background): “Federated Learning aims to train massively distributed models at a large scale (Bonawitz et al., 2019). FedAvg proposed by McMahan et al. (2017) is currently the most widely adopted FL baseline, which reduces communication cost by allowing clients to train multiple iterations locally.”
Better Communication Complexity for Local SGD (2019, background): “Big data optimization problems arising in machine learning and statistics, such as the training of supervised learning models, are routinely solved in a distributed manner on a cluster of compute nodes [2].”
Augment your batch: better training with larger batches (2019, methodology): “Batch computation enables data parallelism (Ben-Nun & Hoeﬂer, 2018), which is necessary to scale training to a large number of processing elements.”
Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks (2019, background): “The modern machine learning paradigm is moving toward parallelization and decentralization [4, 12, 18] to speed up the training and provide reliable solutions to time-sensitive real-world problems.”
SparCML: High-Performance Sparse Communication for Machine Learning (2018, neutral): “There has been a surge of interest in distributed machine learning; see Ben-Nun and Hoeﬂer [5] for a survey.”
Augment Your Batch: Improving Generalization Through Instance Repetition (2020, background): “Batch computation enables data parallelism [2], which is necessary to scale training to a large number of processing elements.”
The future of digital health with federated learning (2020, background): “sks. These strategies are independent of the concept of FL, and can be used in combination with it. The second characteristic of FL - to distribute the compute - has been well studied in recent years [49, 50, 51]. Nowadays, the training of the large-scale models is often executed on multiple devices”
Techniques for Automated Machine Learning (2019, background): “Gradient theorems and approaches [8; 4] have developed several decades, so we would enrich the fundamental theory of AutoDL by shifting the gradient paradigm [8] to AutoDL.”
Understanding Capacity-Driven Scale-Out Neural Recommendation Inference (2020, background): “Training must contend with the accuracy implications of mini-batch sizing and parameter synchronization, and is targeted at maximizing throughput on a single instance that’s fed a large dataset [33].”