Adding noise to the input of a model trained with a regularized objective

By Salah Rifai, Xavier Glorot, Yoshua Bengio, Pascal Vincent. 2011

79 citations recorded by the source corpus.

Regularization is a well studied problem in the context of neural networks. It is usually used to improve the generalization performance when the number of input samples is relatively small or heavily contaminated with noise. The regularization of a parametric model can be achieved in different manners some of which are early stopping (Morgan and Bourlard, 1990), weight decay, output smoothing that are used to avoid overfitting during the training of the considered model. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters (Krogh and Hertz, 1991). Using Bishop's approximation (Bishop, 1995) of the objective function when a restricted type of noise is added to the input of a parametric function, we derive the higher order terms of the Taylor expansion and analyze the coefficients of the regularization terms induced by the noisy input. In particular we study the effect of penalizing the Hessian of the mapping function with respect to the input in terms of generalization performance. We also show how we can control independently this coefficient by explicitly penalizing the Jacobian of the mapping function on corrupted inputs.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistical and Nonlinear Physics

Citations in the corpus

Dropout Training as Adaptive Regularization (2013, methodology): “There is a well-known connection between artificial feature corruption and regularization [8, 9, 10].”
Curriculum Dropout (2017, background): “Several connections [28, 29, 25, 33] have been established between Dropout and model training with noise addition [3, 23, 33].”
Feature Noising for Log-Linear Structured Prediction (2013, methodology): “Sometimes, training with corrupted features reduces to a special form of regularization (Matsuoka, 1992; Bishop, 1995; Rifai et al., 2011b; Wager et al., 2013).”
Deep Learning Made Easier by Linear Transformations in Perceptrons (2012, methodology): “Thirdly, we add randomly generated Gaussian noise to the original input data each time they are presented to the learner, inspired by denoising autoencoders [15] and also recently used in classification [12].”
Invariant representation learning for robust deep networks (2018, background): “This argument has been expanded by others [22, 23, 15].”
Marginalized Denoising Auto-encoders for Nonlinear Representations (2014, methodology): “Rifai et al. (2011b) asks the more direct question: what kind of representations we desire and thus what regularizers do we need for a regular auto-encoder? Their contractive auto-encoder (CAE) thus explicitly encourages learning latent representation to be robust to small perturbation to the inputs”
Analyzing noise in autoencoders and deep networks (2014, methodology): “Alternatively, we can inject additive noise with covariance diag( s (cid:48) f ( Wx ) 2 ) at the hidden unit activations and recover the exact penalty.”
Regularization for Deep Learning: A Taxonomy (2017, methodology): “This term penalizes mappings with large derivatives, and is used in contractive autoencoders (Rifai et al., 2011c).”
Towards Understanding Regularization in Batch Normalization (2018, methodology): “For instance, random noise in the input layer for data augmentation has long been discovered equivalent to a weight decay method, in the sense that the inverse of the signal-to-noise ratio acts as the decay factor (Krogh & Hertz, 1992; Rifai et al., 2011).”