Noisy Activation Functions

By Caglar Gulcehre, Marcin Moczulski, Misha Denil, Yoshua Bengio. 2016

306 citations recorded by the source corpus.

Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order gradients only). Gating mechanisms that use softly saturating activation functions to emulate the discrete switching of digital logic circuits are good examples of this. We propose to exploit the injection of appropriate noise so that the gradients may flow easily, even if the noiseless application of the activation function would yield zero gradient. Large noise will dominate the noise-free gradient and allow stochastic gradient descent toexplore more. By adding noise only to the problematic parts of the activation function, we allow the optimization procedure to explore the boundary between the degenerate (saturating) and the well-behaved parts of the activation function. We also establish connections to simulated annealing, when the amount of noise is annealed down, making it easier to optimize hard objective functions. We find experimentally that replacing such saturating activation functions by noisy variants helps training in many contexts, yielding state-of-the-art or competitive results on different datasets and task, especially when training seems to be the most difficult, e.g., when curriculum learning is necessary to obtain good results.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Pointing the Unknown Words (2016, methodology): “For switching network d t , we observed that us-ing a two-layered MLP with noisy-tanh activation (Gulcehre et al., 2016) function with residual connection from the lower layer (He et al., 2015) activation function to the upper hidden layers improves the BLEU score about 1 points over the d t using…”
Model compression via distillation and quantization (2018, methodology): “This connects quantization to work advocating adding noise to intermediary activations of neural networks as a regularizer, e.g. Gulcehre et al. (2016).”
Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks (2017, background): “It is also quite indispensable for training deep networks [44, 45, 46, 47, 12].”
HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, background): “[32] Caglar Gulcehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio.”
Activation functions in deep learning: A comprehensive survey and benchmark (2021, background): “A noisy AF is defined to overcome the vanishing gradient problem [48].”
Deep relaxation: partial differential equations for optimizing deep neural networks (2017, background): “The latter is common in the deep learning literature under different forms, e.g., Gulcehre et al. (2016) and Mobahi (2016).”
Neural Taylor Approximations: Convergence and Exploration in Rectifier Networks (2016, background): “Two recent papers have shown that noise improves the local optima found during training: Neelakantan et al. (2016) introduce noise into gradients whereas Gulcehre et al. (2016) use noisy activations to extract gradient information from across kinks.”
Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation (2017, background): “Adding noise to sigmoid provides possibilities of training with much wider family of activation functions than previous [10].”
SSDMNV2: A real time DNN-based face mask detection system using single shot multibox detector and MobileNetV2 (2020, unknown): “Most commonly used non-linear functions include different kinds of Rectified Linear Unit (ReLU) (Hahnloser, Sarpeshkar, Mahowald, Douglas, & Seung, 2000), i. e., Leaky ReLU (Zhang, Zou, & Shi, 2017), Noisy ReLU (Gulcehre, Moczulski, Denil, & Bengio, 2016), Exponential ReLU (Opschoor, Schwab, & Zech,”
Estimating Information Flow in Neural Networks (2018, background): “, to escape poor local optima [6], improve generalization performance [7], encourage learning of disentangled representations [8], and ensure gradient flow with hard-saturating nonlinearities [9].”
Analysis on the Dropout Effect in Convolutional Neural Networks (2016, background): “[13] showed that adding carefully chosen noise to the activation can speed up training procedure.”
Accurate deep neural network inference using computational phase-change memory (2019, methodology): “In fact, quantization of the weights or activations 30 , and injecting noise on the weights 31 , activations 32 or gradients 33 have been widely used as DNN regularizers during training to reduce overﬁtting on the training dataset 34,35 .”
Regularization for Deep Learning: A Taxonomy (2017, background): “, 2016b) p(θ) Hidden features Random Training Noisy activation functions (Nair and Hinton, 2010; Xu et al., 2015; Gülçehre et al., 2016a) p(θ|x) Hidden features Random Training”
A survey on modern trainable activation functions (2020, background): “However, the research on classic activation functions has not come to an end, for example, properties on these functions are still discussed in recent studies (Gulcehre et al., 2016; Xu et al., 2016).”
Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes (2016, methodology): “We train our feed-forward controller using noisy-tanh activation function [28] since we were experiencing training difficulties with sigmoid and tanh activation functions.”
Mollifying Networks (2016, background): “At the same time, the impact of noise injection on the behavior of modern deep models has been explored in (Neelakantan et al., 2015) and it has been recently shown that noisy activation functions improve performance on a wide variety of tasks (Gulcehre et al., 2016).”