ImageNet Training in 24 Minutes

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “Increasing interest has focused on large batch training (Goyal et al., 2017; Hoffer et al., 2017; You et al., 2017a), in an attempt to increase the step size and reduce the number of parameter updates required to train a model.”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “The scaling ratio [57] identifies the ratio between computation and communication, wherein DNN models, the computation is proportional to the number of floating point operations required for processing an input while the communication is proportional to model size (or the number of parameters).”
Stochastic Distributed Learning with Gradient Quantization and Variance Reduction (2019, background): “…the computation to communication ratio, such as increased mini-batch sizes (Goyal et al., 2017), deﬁning local problems for each worker (Shamir et al., 2014; Reddi et al., 2016) or reducing the communication frequency (Mc-Donald et al., 2009; Zinkevich et al., 2010; You et al., 2017; Stich, 2018).”
Fast Deep Neural Network Training on Distributed Systems and Cloud TPUs (2019, background): “It is worth noting that our online preprint [16] is”