One weird trick for parallelizing convolutional neural networks

By Alex Krizhevsky. 2014

1,407 citations recorded by the source corpus.

I present a new way to parallelize the training of convolutional neural networks across multiple GPUs. The method scales significantly better than all alternatives when applied to modern convolutional neural networks.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, methodology): “We adopt “data parallelism” [17] on the conv layers.”
Convolutional neural networks at constrained time cost (2014, methodology): “We implement the models using the cuda-convnet2 library3 [13] with our modifications.”
Very Deep Convolutional Networks for Large-Scale Image Recognition (2014, methodology): “While more sophisticated methods of speeding up ConvNet tra ining have been recently proposed [9], which employ model and data parallelism for different layer s of the net, we have found that our conceptually much simpler scheme already provides a speedu of 3.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “, 2018] for ease of implementation of both model parallelism and data parallelism [Krizhevsky, 2014].”
Beyond human-level accuracy: computational challenges in deep learning (2019, background): “Prior work also explores other forms of data parallelism [20, 23, 28], and there are opportunities to reduce data communication overheads of parallelism [5, 21, 37].”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “theory suggesting that an E-times increase in batch size should be accompanied by a √ E increase in learning rate (Krizhevsky, 2014).”
Don't Decay the Learning Rate, Increase the Batch Size (2017, methodology): “To our knowledge, this scaling rule was ﬁst adopted by Krizhevsky (2014).”
TensorFlow: A system for large-scale machine learning (2016, neutral): “Single-machine frameworks Many machine learning researchers carry out their work on a single—often GPU-equipped—computer [41, 42], and many ﬂexible single-machine frameworks have emerged to support this scenario.”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “In practice, the learning rate is often scaled when the batch size increases [26, 16].”
Large Batch Training of Convolutional Networks (2017, methodology): “The brute-force way to speed up CNN training is to add more computational power (e.g. more GPU nodes) and train network using data-parallel Stochastic Gradient Descent, where each worker receives some chunk of global mini-batch (see e.g. Krizhevsky (2014) or Goyal et al. (2017) ).”
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017, methodology): “The above linear scaling rule was adopted by Krizhevsky [21], if not earlier.”
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “The Squeezenet, AlexNet, and VGG architectures start at 64 .”
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016, methodology): “Some TensorFlow uses allow some ﬂexibil-ity in terms of the consistency of parameter updates, and we can easily express and take advantage of these relaxed synchronization requirements in some of our larger de-ployments.”
MegDet: A Large Mini-Batch Object Detector (2017, methodology): “Previous works [21, 13, 39] use Linear Scaling Rule, which changes the new learning rate to r̂ ← k · r.”
The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning (2017, methodology): “A number of recent large scale neural network methods including (Krizhevsky, 2014; Chen et al., 2016; Goyal et al., 2017) use the “linear scaling rule” to accelerate training using parallel computation.”
Visual Prompt Tuning (2022, positive): “Following the linear scaling rule [38,23,9,26], the learning rate is set as”
Distributed Prioritized Experience Replay (2018, background): “The resulting parameter updates may be applied synchronously (Krizhevsky, 2014) or asynchronously (Dean et al.”