Efficient mini-batch training for stochastic optimization

By Mu Li, T. Zhang, Yuqiang Chen, Alex Smola. 2014

848 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “Other optimizers, such as KFAC (Martens and Grosse, 2015; Grosse and Martens, 2016; Ba et al., 2017), or optimization techniques designed speciﬁcally for massively data parallel systems (e.g. Li et al., 2014), might allow perfect scaling to extend much further.”
TensorFlow: A system for large-scale machine learning (2016, neutral): “Many scalable algorithms train a model using mini-batch gradient descent [21, 47], where a worker reads the current version of the model and a small batch of input examples, calculates an update to the model that reduces a loss function on those examples, and applies the update to the model.”
Large Batch Training of Convolutional Networks (2017, methodology): “But using large batch may negatively impact the model accuracy, as was observed in Krizhevsky (2014), Li et al. (2014), Keskar et al. (2016), Hoffer et al. (2017),.. Increasing the global batch while keeping the same number of epochs means that you have fewer iterations to update weights.”
The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning (2017, background): “There have been some attempts to understand the interplay between the mini-batch size and computational efﬁciency, e.g., [TBRS13, LZCS14] in the standard non-interpolated regime.”
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016, background): “In (Li et al., 2014), the authors argue that the convergence rate of SGD for the large-batch setting can be improved by obtaining iterates through the following proximal sub-problem. x k +1 = arg min (5) The motivation for this strategy is, in the context of large-batch methods, to better utilize a…”
Federated Optimization in Heterogeneous Networks (2018, methodology): “…proximal term we introduce has also been explored in previous optimization literature with different purposes, such as Allen-Zhu (2018), to speed up (mini-batch) SGD training on a single machine, and in Li et al. (2014b) for efficient SGD training both in a single machine and distributed settings.”
Is Local SGD Better than Minibatch SGD? (2020, neutral): “(Zinkevich et al., 2010; Zhang et al., 2012; Li et al., 2014; Rosenblatt and Nadler, 2016; Godichon-Baggioni and Saadane, 2017; Jain et al., 2017).”
Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some of the notable references in a chronological order are Langford et al. (2009); Niu et al. (2011); Agarwal and Duchi (2011); Paine et al. (2013); Li et al. (2014b); Zhang et al. (2014); Keuper and Pfreundt (2015); De and Goldstein (2015); Feyzmahdavian et al. (2015); Lian et al. (2015); Mania…”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “It has been shown that training with large batch size (e.g. batch size > 103 for the case of ImageNet) typically degrades the performance both in terms of training and test error (Goyal et al., 2017; Chen & Huo, 2016; Hoffer et al., 2017; Keskar et al., 2016; Li, 2017; Li et al., 2014).”
Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD (2018, neutral): “Smaller mini-batches result in higher variance of the gradients, which affects convergence and error floor [Bottou et al., 2016, Dekel et al., 2012, Li et al., 2014].”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “The aim of mini-batch gradient descent (Li et al., 2014)is to solve the following two problems.”
Classification-based financial markets prediction using deep neural networks (2016, methodology): “It is well known that mini-batching improves the computational performance of the feedforward and back-propagation computations (Shekhar and Amin, 1994; Li et al., 2014).”
Opportunities and obstacles for deep learning in biology and medicine (2017, background): “The massively parallel nature of GPUs allows additional optimizations, such as accelerated mini-batch gradient descent [379,380,383,384].”
Gradient Sparsification for Communication-Efficient Distributed Optimization (2017, background): “The existing works on distributed machine learning mainly focus on two directions: 1) how to design communication efficient algorithms to reduce the round of communications among workers [43, 31, 13, 27, 42], and 2) about how to use large mini-batches without compromising the convergence speed [21, ”
Stochastic Variance Reduction for Nonconvex Optimization (2016, background): “In contrast, Sgd does not yield an eﬃcient mini-batch strategy as it requires O ( b 1 / 2 / 2 ) IFO calls for achieving an (cid:15) -accurate solution (Li et al., 2014).”
Memory and Communication Efficient Distributed Stochastic Optimization with Minibatch Prox (2017, methodology): “In Li et al. (2014), the authors proposed a simple algorithm EMSO to approximately solve (15), where each machine first solve its own local objective, i.e., w (i) t = argmin w∈Ω φ I (i) t + γ 2 ‖w −wt−1‖2 , (16) and then all machines average their local solutions via one round of communication: wt…”
Sub-Sampled Newton Methods I: Globally Convergent Algorithms (2016, background): “Consequently, stochastic variant of full gradient descent, e.g., (mini-batch) stochastic gradient descent (SGD) was developed [39, 8, 26, 5, 7, 14].”
AdaBatch: Efficient Gradient Aggregation Rules for Sequential and Parallel Stochastic Gradient Methods (2017, background): “In [14], a specific subproblem is solved instead of just averaging the gradients in a mini-batch.”
Cryptocurrency portfolio management with deep reinforcement learning (2016, methodology): “The fact that the input of the agent is independent of the last output, allows us to use the mini-batch training to speed up computation [23].”