Training Recurrent Neural Networks by Diffusion

By Hossein Mobahi. 2016

46 citations recorded by the source corpus.

This work presents a new algorithm for training recurrent neural networks (although ideas are applicable to feedforward networks as well). The algorithm is derived from a theory in nonconvex optimization related to the diffusion equation. The contributions made in this work are two fold. First, we show how some seemingly disconnected mechanisms used in deep learning such as smart initialization, annealed learning rate, layerwise pretraining, and noise injection (as done in dropout and SGD) arise naturally and automatically from this framework, without manually crafting them into the algorithms. Second, we present some preliminary results on comparing the proposed method against SGD. It turns out that the new algorithm can achieve similar level of generalization accuracy of SGD in much fewer number of epochs.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

Citations in the corpus

Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “Other ideas which also penalize sharp minima include operating on diffused loss landscape (Mobahi, 2016) and regularizing local entropy (Chaudhari et al., 2016).”
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016, methodology): “Indeed, similar algorithms were proposed in (Zhang et al., 2015) and (Mobahi, 2016) for Deep Learning.”
Entropy-SGD: biasing gradient descent into wide valleys (2016, methodology): “Let us remark that methods in the deep learning literature that average the gradient through perturbations in the neighborhood of x (Mobahi, 2016) or noisy activation functions (Gulcehre et al., 2016) can be interpreted as computing the data likelihood in ELBO (without the KL-term); such an…”
Noisy Networks for Exploration (2017, background): “We need to emphasise that the idea of injecting noise to improve the optimisation process has been thoroughly studied in the literature of supervised learning and optimisation under different names (e.g., Neural diffusion process (Mobahi, 2016) and graduated optimisation (Hazan et al., 2016)).”
ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks (2021, background): “…generalization bounds, loss surface sharpness and minimization of the derived generalization bound have proven to be effective in attaining state-of-the-art performances in various tasks (Hochreiter & Schmidhuber, 1997; Mobahi, 2016; Chaudhari et al., 2019; Sun et al., 2020; Yue et al., 2020).”
Deep relaxation: partial differential equations for optimizing deep neural networks (2017, background): “The latter is common in the deep learning literature under different forms, e.g., Gulcehre et al. (2016) and Mobahi (2016).”
The Effect of Gradient Noise on the Energy Landscape of Deep Networks (2015, methodology): “Conceptually, our approach is similar to homotopy continuation methods (Allgower and Georg, 2012; Mobahi, 2016) which start optimization with a benign energy landscape and gradually recover the original problem.”
SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning (2018, methodology): “This reveals a pitfall in implementation that the noise θ t added to w t must be denoised before applying the gradient, which is also a key difference from existing noise injection approaches [21][22][23][24].”
Mollifying Networks (2016, methodology): “This is related to the work of Mobahi (2016), who recently introduced analytic smooths of neural network non-linearities in order to help training recurrent networks.”
Noisy Activation Functions (2016, methodology): “Recently, Mobahi (2016) propose an optimization method that applies Gaussian smoothing on the loss function such that annealing weight noise is a Monte Carlo estimator of that.”
Kickstarting Deep Reinforcement Learning (2018, methodology): “Then through adaptation of λ k during the course of learning, the agent is able to shift its optimization focus on the (potentially sparse) reward signal r t , similar to how continuation methods relax optimisation problems to make ﬁnding the solution easier (Gulcehre et al., 2016; Mobahi, 2016).”
On the energy landscape of deep networks (2015, methodology): “Local minima proliferate the energy landscape: there are exponentially many of them, at all energy levels.”