Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering

By Kai Chen, Qiang Huo. 2016

146 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017, background): “Training techniques to overcome these effects are beyond the scope of this work ( e.g ., quantized gradients [18], Block-Momentum SGD [6]).”
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, methodology): “We also use Blockwise Momentum Update Filtering (BMUF) [19] for all the experiments to help with scaling the training workflows.”
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, methodology): “Inspired by BMUF, we propose a general framework called slow momentum (SLOWMO) to improve the accuracy of communication-efficient distributed training methods.”
Don't Use Large Mini-Batches, Use Local SGD (2018, methodology): “It has been shown that training with large batch size (e.g. batch size > 103 for the case of ImageNet) typically degrades the performance both in terms of training and test error (Goyal et al., 2017; Chen & Huo, 2016; Hoffer et al., 2017; Keskar et al., 2016; Li, 2017; Li et al., 2014).”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, background): “Consider the distributed training of deep neural networks over multiple workers (Dean et al. 2012), where all workers can access all or partial training data and aim to find a common model that yields the minimum training loss.”
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD (2018, methodology): “To combine momentum and periodic averaging, we use the block momentum scheme that was proposed in [3, 21] and applied to speech recognition tasks.”
Lessons from Building Acoustic Models with a Million Hours of Speech (2019, methodology): “We explored the Gradient Threshold Compression method (GTC) [35] and Blockwise Model-Update Filtering (BMUF) [41].”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “Such methods may separately run m SGD instances on different machines, aggregating the parameters only once (post-training) [277] or every few iterations [33, 174].”
On the Convergence of Local Descent Methods in Federated Learning (2019, background): “Empirical advantages of model averaging is studied in [5, 13, 23, 25,27, 32].”
PopSGD: Decentralized Stochastic Gradient Descent in the Population Model (2019, background): “[24, 53] and [19, 45]), or even decentralized synchronous variants(e.”
Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features (2016, methodology): “To speed up the experiments, we have exploited the block-wise model-update ﬁltering (BMUF) distributed training algorithm implemented in CNTK, originally proposed by Chen and Huo [2].”
Parallel Restarted SGD for Non-Convex Optimization with Faster Convergence and Less Communication (2018, background): “Various experimental results [12, 1, 11, 14, 5] observe that model averaging can provide superior performance for deep learning applications.”
Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks (2016, methodology): “– MA-DNN refers the parallel DNN training framework with the aggregation by averaging model parameters [27, 20, 6, 26, 2, 3].”
A Survey on Distributed Machine Learning (2019, background): “• Block-momentum SGD (Chen and Huo [31]) divides the training set into m blocks and n splits.”
From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition (2019, unknown): “875 block momentum [43].”
Asynchronous Stochastic Gradient Descent with Delay Compensation (2016, background): “Copyright 2017 by the author(s). et al., 2015; Chen & Huo, 2016; Chen et al., 2016).”