Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities

By Tommi Vatanen, Tapani Raiko, Harri Valpola, Yann LeCun. 2013

34 citations recorded by the source corpus.

Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scale of the outputs of each hidden neuron, and secondly by analyzing the connections to second order optimization methods. We show that the transformations make a simple stochastic gradient behave closer to second-order optimization methods and thus speed up learning. This is shown both in theory and with experiments. The experiments on the third transformation show that while it further increases the speed of learning, it can also hurt performance by converging to a worse local optimum, where both the inputs and outputs of many hidden neurons are close to zero.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, methodology): “The papers of [39, 38, 31, 47] propose methods for centering layer responses, gradients, and propagated errors, implemented by shortcut connections.”
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, methodology): “In [20, 25, 35] the analysis of FIM was a motivation of centering the nonlinear responses of each layer, which eases training of symmetric, sigmoid-like units.”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “Centering methods work by either modifying the gradient (Schraudolph, 1998) or dynamically reparameterizing the network itself (Raiko et al., 2012; Vatanen et al., 2013; Wiesler et al., 2014), so that various unit-wise scalar quantities like the activities (the a i ’s) and local derivatives (the φ…”
Natural Neural Networks (2015, background): “[28] extended their previous work [17] by introducing a multiplicative constant i”
Knowledge Matters: Importance of Prior Information for Optimization (2013, background): “Centering the activations of each feature detector in a neural network has been studied in (Raiko et al., 2012) and (Vatanen et al., 2013).”
Kronecker-factored Curvature Approximations for Recurrent Neural Networks (2018, result): “Motivated in part by these older results and by the more recent success of centering and normalization methods (e.g. Schraudolph, 1998; Vatanen et al., 2013; Ioffe & Szegedy, 2015) a new family of methods has emerged that are based on non-diagonal curvature matrix approximations the rely on the…”
Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters (2015, background): “However, arguably, batch normalization (Ioffe and Szegedy, 2015) is eliminating part of the problem, by making the Hessian closer to identity (Vatanen et al., 2013; Raiko et al., 2012), making the approximation more justified.”
ImageNet classification with deep convolutional neural networks (2012, unknown): “The papers of [39, 38, 31, 47] propose methods for centering layer responses, gradients, and propagated errors, implemented by shortcut connections.”