Deep Learning Made Easier by Linear Transformations in Perceptrons

By Tapani Raiko, Harri Valpola, Yann LeCun. 2012

209 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, methodology): “The papers of [39, 38, 31, 47] propose methods for centering layer responses, gradients, and propagated errors, implemented by shortcut connections.”
Training Very Deep Networks (2015, background): “Skip connections between layers or to output layers (where error is “injected”) have long been used in neural networks, more recently with the explicit aim to improve the flow of information [22, 23, 2, 24].”
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, methodology): “In [20, 25, 35] the analysis of FIM was a motivation of centering the nonlinear responses of each layer, which eases training of symmetric, sigmoid-like units.”
Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities (2013, methodology): “The goal of Equation (5) is to normalize both the output signals (similarly as data is often normalized as a preprocessing step – see,e.g., [8]) and the slopes of the output signals of each hidden unit at the same time.”
Wide Residual Networks (2016, background): “We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts.”
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, methodology): “For example, Raiko et al. [23] propose to transform the outputs of each neuron to have zero output and zero slope on average.”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “Also the Projected Natural Gradient Descent algorithm (PRONG) centers the activations by implicitly whitening them (Desjardins et al., 2015).”
Representation Learning: A Review and New Perspectives (2012, background): “Another promising idea to improve the conditioning of neural network training is to nullify the average value and slope of each hidden unit output [158], and possibly locally”
How to Construct Deep Recurrent Neural Networks (2013, background): “One possible way to address this difﬁculty is to introduce shortcut connections (see, e.g., Raiko et al. , 2012) An approach similar to the deep hidden-to-hidden transition has been proposed recently by Pinheiro and Collobert (2014) in the context of parsing a static scene.”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “It is argued by Raiko et al. (2012) that the application of the centering transformation makes the Fisher of the resulting network closer to a diagonal matrix, and thus makes its gradient more closely resemble its natural gradient.”
Deep learning in neural networks: An overview (2014, background): “DL in NNs can also be improved by transforming hidden unit activations such that they have zero output and slope on average (Raiko et al., 2012).”
Measuring and regularizing networks in function space (2018, background): “It appears that many approaches to normalize and whiten activations or gradients have been forwarded in the literature (Raiko et al. (2012);Simard et al. (1998); Schraudolph & Sejnowski (1996); Crammer et al. (2009); Wang et al. (2013); LeCun et al. (1991); Schraudolph (1998); Salimans & Kingma…”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “A recent heuristic that is related to the diﬃculty of gradient propagation through neural net non-linearities is the idea of “centering” the non-linear operation such that each hidden unit has zero average output and zero average slope (Schraudolph, 1998; Raiko et al. , 2012).”
Natural Neural Networks (2015, methodology): “Inspired by recent work on model reparametrizations [17, 13], our approach starts with a simple question: can we devise a neural network architecture whose Fisher is constrained to be identity?”
Techniques for Learning Binary Stochastic Feedforward Neural Networks (2014, methodology): “4 was added to each pixel separately in each epoch (Raiko et al., 2012).”
Deep Learning of Representations: Looking Forward (2013, background): “A related idea that may have an impact on ill-conditioning is the idea of skip-connections, which forces both the mean output and the mean slope of each hidden unit of a deep multilayer network to be zero (Raiko et al., 2012), a centering idea which originates from Schraudolph (1998).”
Architectural Complexity Measures of Recurrent Neural Networks (2016, background): “Some examples include Raiko et al. (2012); Graves (2013) who explored the use of skip connections; Hermans & Schrauwen (2013) who proposed the deep RNNs which are stacked RNNs with skip connections; Pascanu et al. (2013a) who pointed out the distinction of constructing a “deep” RNN from the view of…”
Knowledge Matters: Importance of Prior Information for Optimization (2013, background): “Centering the activations of each feature detector in a neural network has been studied in (Raiko et al., 2012) and (Vatanen et al., 2013).”