Centering Neural Network Gradient Factors

By N. Schraudolph. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, methodology): “The papers of [39, 38, 31, 47] propose methods for centering layer responses, gradients, and propagated errors, implemented by shortcut connections.”
Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities (2013, background): “In learning multi-layer perceptron (MLP) networks by back-propagation, there are known transformations that speed up learning [8, 11, 12].”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, positive): “L G ] 2 2 Fe b 20 16 the activations speeds up learning (LeCun et al., 1991; 1998; Schraudolph, 1998).”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “A recent heuristic that is related to the diﬃculty of gradient propagation through neural net non-linearities is the idea of “centering” the non-linear operation such that each hidden unit has zero average output and zero average slope (Schraudolph, 1998; Raiko et al. , 2012).”
Deep Learning of Representations: Looking Forward (2013, background): “idea of skip-connections, which forces both the mean output and the mean slope of each hidden unit of a deep multilayer network to be zero (Raiko et al., 2012), a centering idea which originates from Schraudolph (1998). There has also been very successful recent work exploiting rectiﬁer non-linearit”
Metric-Free Natural Gradient for Joint-Training of Boltzmann Machines (2013, background): “Since, Montavon and Muller [9] have shown that this difficulty stems from an ill-conditioning of the Hessian matrix, which can be addressed by a simple reparameterization of the DBM energy function, a trick called centering (an analogue to centering and skip-connections found in the deterministic ne”
Diagonal Rescaling For Neural Networks (2017, methodology): “To that effect, we consider reparametrizations of feedforward neural networks that are closely connected to the normalization of neural network activations [Schraudolph, 2012, Ioffe and Szegedy, 2015] and are amenable to zero overhead stochastic gradient implementations.”
ImageNet classification with deep convolutional neural networks (2012, unknown): “The papers of [39, 38, 31, 47] propose methods for centering layer responses, gradients, and propagated errors, implemented by shortcut connections.”
Kronecker-factored Curvature Approximations for Recurrent Neural Networks (2018, result): “Motivated in part by these older results and by the more recent success of centering and normalization methods (e.g. Schraudolph, 1998; Vatanen et al., 2013; Ioffe & Szegedy, 2015) a new family of methods has emerged that are based on non-diagonal curvature matrix approximations the rely on the…”
Deep Learning Made Easier by Linear Transformations in Perceptrons (2012, background): “In learning multi-layer perceptron (MLP) networks by back-propagation, there are known transformations that speed up learning [10] [13] [14].”