Proposal Scaling Distributed Machine Learning with System and Algorithm Co-design

By Mu Li. 2016

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning (2017, methodology): “√ To keep the same covariance, [BCN16, Li17, HHS17] choose the step size η ∼ m for batch size m .”
Train longer, generalize better: closing the generalization gap in large batch training of neural networks (2017, neutral): “The convergence of SGD is also known to be affected by the batch size (Li et al., 2014), but in this work we will focus on generalization. Most of our results were conducted on the Resnet44 topology, introduced by He et al. (2016). We strengthen our findings with additional empirical results in sect”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “Thus, several prior works carefully hand-tune training hyper-parameters, like learning rate and momentum, to avoid degradation of generalization performance (Goyal et al., 2017; Li, 2017; You et al., 2018; Shallue et al., 2018).”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “It has been shown that training with large batch size (e.g. batch size > 103 for the case of ImageNet) typically degrades the performance both in terms of training and test error (Goyal et al., 2017; Chen & Huo, 2016; Hoffer et al., 2017; Keskar et al., 2016; Li, 2017; Li et al., 2014).”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, background): “To overcome this problem, we increase the learning rate for large-batch scenario [10].”
ResNeSt: Split-Attention Networks (2020, methodology): “2 For effectively training deep CNN models, we follow the prior work [18, 34, 36] to train our models using 8 servers (64 GPUs in total) in parallel.”
Scaling SGD Batch Size to 32K for ImageNet Training (2017, methodology): “On the other hand, Goyal et al [7], Krizhevsky et al [13], and Li et al [15] use empirical study to show that controlling the learning rate in a smart way can help the large-batch training to get the same accuracy with the baseline.”
ImageNet Training in 24 Minutes (2017, unknown): “Team Model Baseline Batch Large Batch Baseline Accuracy Large Batch Accuracy Google (Krizhevsky 2014) AlexNet 128 1024 57.7% 56.7% Amazon (Li 2017) ResNet-152 256 5120 77.8% 77.8% Facebook (Goyal et al. 2017) ResNet-50 256 8192 76.40% 76.26% (a) Batch Size=16k (b) Batch Size=32k Figure 3: The base l”
A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication (2018, background): “In this context, a well-known performance bottleneck is the high bandwidth cost for synchronizing the gradients or model parameters among multiple machines [3, 27, 20, 28, 9, 42, 29].”
Reducing BERT Pre-Training Time from 3 Days to 76 Minutes (2019, background): “The researchers needed to carefully tune training hyper-parameters, like learning rate and momentum, to avoid losing test accuracy (Goyal et al., 2017; Li, 2017; You et al., 2018; Shallue et al., 2018).”