Integrated Model, Batch, and Domain Parallelism in Training Neural Networks

By A. Gholami, Ariful Azad, Peter H. Jin, Kurt Keutzer, Aydin Buluç. 2017

88 citations recorded by the source corpus.

We propose a new integrated method of exploiting model, batch and domain parallelism for the training of deep neural networks (DNNs) on large distributed-memory computers using minibatch stochastic gradient descent (SGD). Our goal is to find an efficient parallelization strategy for a fixed batch size using P processes. Our method is inspired by the communication-avoiding algorithms in numerical linear algebra. We see P processes as logically divided into a P_r x P_c grid where the P_r dimension is implicitly responsible for model/domain parallelism and the P_c dimension is implicitly responsible for batch parallelism. In practice, the integrated matrix-based parallel algorithm encapsulates these types of parallelism automatically. We analyze the communication complexity and analytically demonstrate that the lowest communication costs are often achieved neither with pure model nor with pure data parallelism. We also show how the domain parallel approach can help in extending the theoretical scaling limit of the typical batch parallel method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Mesh-TensorFlow: Deep Learning for Supercomputers (2018, methodology): “[18] analytically showed that using both data and model parallelism at the same time can be more beneficial than using just one of them.”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “1× speedup was achieved for 4 GPUs using the same approach, and derived theoretically using communication cost analysis [80], promoting 1.”
OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks (2022, background): “Furthermore, partitioning a deep neural network efficiently to balance communication and computation remains an open problem still actively researched [31, 44, 55].”
Spatially Parallel Convolutions (2018, background): “The combination of spatial parallelism with both data and model parallelism (Gholami et al., 2018) is a design point, for which an optimum exists that minimizes the sum cost of computation and communication.”
Data Movement Is All You Need: A Case Study on Optimizing Transformers (2020, methodology): “Other optimizations, including model parallelism (Van Es-sen et al., 2015; Jia et al., 2019d; Gholami et al., 2018; Dean et al., 2012; Chilimbi et al., 2014; Shazeer et al., 2018; Buchlovsky et al., 2019), pipeline parallelism (Chen et al., 2012; Li et al., 2018; Narayanan et al., 2019; Huang et…”