Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability

By Janis Keuper, Franz-Josef Pfreundt. 2016

104 citations recorded by the source corpus.

This paper presents a theoretical analysis and practical evaluation of the main bottlenecks towards a scalable distributed solution for the training of Deep Neural Networks (DNNs). The presented results show, that the current state of the art approach, using data-parallelized Stochastic Gradient Descent (SGD), is quickly turning into a vastly communication bound problem. In addition, we present simple but fixed theoretic constraints, preventing effective scaling of DNN training beyond only a few dozen nodes. This leads to poor scalability of DNN training in most practical scenarios.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Distributed Deep Reinforcement Learning: Learn how to play Atari games in 21 minutes (2018, background): “This is because in a gradient descent training the communication cannot be easily overlaid with computation to maximally utilize both network bandwidth and compute power (see [17] for details).”
Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020, background): “…Learning is a related ﬁeld with similarities and key differences: while both study the theory and practice involving partitioning of data and aggregation of model updates [19, 9, 37, 70], federated learning is additionally concerned with data that is private and distributed in a non-i.i.d. fashion.”
Efficient sparse collective communication and its application to accelerate distributed deep learning (2020, unknown): “Indeed, the fact that communication is a major performance bottleneck in DDL is well-known [32], and many works [10, 35, 39, 44, 58, 66] proposed various optimizations to achieve high-bandwidth collective communication specialized for DDL.”