Accelerated Gradient Descent by Factor-Centering Decomposition

By N. Schraudolph. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, methodology): “The papers of [39, 38, 31, 47] propose methods for centering layer responses, gradients, and propagated errors, implemented by shortcut connections.”
Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities (2013, background): “In learning multi-layer perceptron (MLP) networks by back-propagation, there are known transformations that speed up learning [8, 11, 12].”
Measuring and regularizing networks in function space (2018, background): “…that many approaches to normalize and whiten activations or gradients have been forwarded in the literature (Raiko et al. (2012);Simard et al. (1998); Schraudolph & Sejnowski (1996); Crammer et al. (2009); Wang et al. (2013); LeCun et al. (1991); Schraudolph (1998); Salimans & Kingma (2016)).”
Deep Learning Made Easier by Linear Transformations in Perceptrons (2012, background): “In learning multi-layer perceptron (MLP) networks by back-propagation, there are known transformations that speed up learning [10] [13] [14].”
Decorrelated Batch Normalization (2018, background): “Normalized activations [35, 51] and gradients [44, 38] have long been known to be beneficial for training neural networks.”
ImageNet classification with deep convolutional neural networks (2012, unknown): “The papers of [39, 38, 31, 47] propose methods for centering layer responses, gradients, and propagated errors, implemented by shortcut connections.”
Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, background): “Normalized activations [38, 33, 31, 46] have long been known to benefit neural networks training.”