Gradient Diversity Empowers Distributed Learning

By Dong Yin, Ashwin Pananjady, Maximilian Lam, Dimitris Papailiopoulos, Kannan Ramchandran, Peter L. Bartlett. 2017

12 citations recorded by the source corpus.

It has been experimentally observed that distributed implementations of mini-batch stochastic gradient descent (SGD) algorithms exhibit speedup saturation and decaying generalization ability beyond a particular batch-size. In this work, we present an analysis hinting that high similarity between concurrently processed gradients may be a cause of this performance degradation. We introduce the notion of gradient diversity that measures the dissimilarity between concurrent gradient updates, and show its key role in the performance of mini-batch SGD. We prove that on problems with high gradient diversity, mini-batch SGD is amenable to better speedups, while maintaining the generalization performance of serial (one sample) SGD. We further establish lower bounds on convergence where mini-batch SGD slows down beyond a particular batch-size, solely due to the lack of gradient diversity. We provide experimental evidence indicating the key role of gradient diversity in distributed learning, and discuss how heuristics like dropout, Langevin dynamics, and quantization can improve it.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Ray Interference: a Source of Plateaus in Deep Reinforcement Learning (2019, neutral): “Additionally, there are methods that prevent the interference directly at the gradient level [5, 47], normalize the scales of the losses [15], or explicitly preserve capacity for late-learned subtasks [18].”
Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks (2017, background): “Gradient diversity: Yin et al. (2017) construct a scalar measure of the gradient diversity given by ∑k‖∇ fk(x)‖/‖∇ f (x)‖, and analyze its effect on the maximum allowed batch-size in the context of distributed optimization.”
Federated Learning with Buffered Asynchronous Aggregation (2021, background): “It is well accepted that increasing concurrency eventually saturates beyond certain batch size in synchronized methods (Yin et al., 2017; Ott et al., 2018; Goyal et al., 2017; Ott et al., 2018; You et al., 2019; 2018; 2017; Shallue et al., 2018).”