Mean-normalized stochastic gradient for large-scale deep learning

By Simon Wiesler, Alexander Richard, Ralf Schlüter, Hermann Ney. 2014

74 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, methodology): “…whitening activations at every training step or at some interval, either by modifying the network directly or by changing the parameters of the optimization algorithm to depend on the network activation values (Wiesler et al., 2014; Raiko et al., 2012; Povey et al., 2014; Desjardins & Kavukcuoglu).”
YouTube-8M: A Large-Scale Video Classification Benchmark (2016, background): “Standardization of features has been proven to help with online learning algorithms[14, 37] as it makes the updates using Stochastic Gradient Descent (SGD) based algorithms (like Adagrad) more robust to learning rates, and speeds up convergence.”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “…by either modifying the gradient (Schraudolph, 1998) or dynamically reparameterizing the network itself (Raiko et al., 2012; Vatanen et al., 2013; Wiesler et al., 2014), so that various unit-wise scalar quantities like the activities (the ai’s) and local derivatives (the φ′i(si)’s) are 0 on…”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, methodology): “Finally, low-rank factorizations and sketches of the gradients when viewed as matrices were proposed in [35, 36, 37, 38, 12]; arguably most of these methods (with the exception of [12]) aimed to address the high flops required when training low-rank models.”
RASR/NN: The RWTH neural network toolkit for speech recognition (2014, methodology): “While the binary format is more efficient, the XML format makes it easy to debug and analyze models as well as to import models from other neural network software.”
Double Forward Propagation for Memorized Batch Normalization (2018, background): “To alleviate Internal Covariate Shift , one can whiten the output of each layer, but it turns out to be very computationally expensive (Wiesler et al. 2014; Raiko, Valpola, and LeCun 2012).”
Deep Residual Bidir-LSTM for Human Activity Recognition Using Wearable Sensors (2017, background): “Such a small standard deviation is often useful in deep learning [32].”
Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks (2018, unknown): “…obvious idea is to train neural networks with this same structure– effectively, with linear bottlenecks– from a random start; but when this has been tried in the past, the stability of training has been a problem [3], although there have been other reports of successfully training such models [4].”
A comprehensive study of deep bidirectional LSTM RNNS for acoustic modeling in speech recognition (2016, methodology): “We compare stochastic gradient descent (SGD), SGD with momentum [32, 33] where one variant only depends on the last minibatch (mom) and another variant depends on the full history (mom2), SGD with Nesterov momentum [34, 33], mean-normalized SGD (MNSGD) [35], Adadelta [36], Adagrad [37], Adam and Ada”
Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, methodology): “Some of these methods include activations centering in Restricted Boltzmann Machine [31]/feed-forward neural networks [46] and activations whitening [9, 29].”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “[49] also considered centering the activations to improve the performance of DNNs, reformulating the centering normalization by re-parameterization.”
Acoustic modeling with deep neural networks using raw time signal for LVCSR (2014, unknown): “introducing narrow matrix factorization layers [7].”
Advances in Very Deep Convolutional Neural Networks for LVCSR (2016, unknown): “Batch normalization (BN) was introduced in [4], and is closely related to prior work aimed at whitening the activations inside the network [30].”