The Effect of Gradient Noise on the Energy Landscape of Deep Networks

By Pratik Chaudhari, Stefano Soatto. 2015

16 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Entropy-SGD: biasing gradient descent into wide valleys (2016, background): “…networks has been predicted using a wide variety of theoretical analyses and empirical observations, e.g., papers such as Choro-manska et al. (2015a;b); Chaudhari & Soatto (2015) that build upon results from statistical physics as also others such as Haeffele & Vidal (2015) and Janzamin et al.…”
Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks (2017, background): “There is however a disconnect between these two directions due to the fact that while adding external gradient noise helps in theory, it works poorly in practice (Neelakantan et al., 2015; Chaudhari and Soatto, 2015).”
Deep relaxation: partial differential equations for optimizing deep neural networks (2017, background): “Based on empirical results such as Dauphin et al. (2014), authors in Bray and Dean (2007); Fyodorov and Williams (2007); Choromanska et al. (2015) and Chaudhari and Soatto (2015) have modeled the energy landscape of deep neural networks as a high-dimensional Gaussian random ﬁeld.”
Regularization for Deep Learning: A Taxonomy (2017, background): “Wilson et al. (2017) show that some of the methods such as AdaGrad or Adam even lose the regularization abilities of SGD. • Update rules – Momentum, Nesterov’s accelerated gradient method, AdaGrad, AdaDelta, RMSProp, Adam—overview in (Wilson et al., 2017) – Learning rate schedules (Girosi et al., 19”