Mollifying Networks

By Caglar Gulcehre, Marcin Moczulski, Francesco Visin, Yoshua Bengio. 2016

47 citations recorded by the source corpus.

The optimization of deep neural networks can be more challenging than the traditional convex optimization problems due to highly non-convex nature of the loss function, e.g. it can involve pathological landscapes such as saddle-surfaces that can be difficult to escape from for algorithms based on simple gradient descent. In this paper, we attack the problem of optimization of highly non-convex neural networks objectives by starting with a smoothed -- or mollified -- objective function which becomes more complex as the training proceeds. Our proposition is inspired by the recent studies in continuation methods: similarly to curriculum methods, we begin by learning an easier (possibly convex) objective function and let it evolve during training until it eventually becomes the original, difficult to optimize objective function. The complexity of the mollified networks is controlled by a single hyperparameter that is annealed during training. We show improvements on various difficult optimization tasks and establish a relationship between recent works on continuation methods for neural networks and mollifiers.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Entropy-SGD: biasing gradient descent into wide valleys (2016, methodology): “…convolve the loss function to solve sequentially reﬁned optimization problems (Allgower & Georg, 2012; Mobahi & Fisher III, 2015), similarly, methods that perturb the weights or activations to average the gradient (Gulcehre et al., 2016) do so with an aim to smooth the rugged energy landscape.”
Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, background): “To alleviate the local minimum problem during training, other techniques that smooth the loss function have been proposed recently [8, 15].”
C-MIL: Continuation Multiple Instance Learning for Weakly Supervised Object Detection (2019, background): “In [20], “molliﬁers” were introduced to smooth the loss function by gradually increasing the difﬁculty of the optimization problem.”
Online Deep Learning: Learning Deep Neural Networks on the Fly (2017, methodology): “…that shallower models tend to converge faster than deeper models (Chen, Goodfellow, and Shlens 2016; Larsson, Maire, and Shakhnarovich 2016; Gulcehre et al. 2016), using a hedging strategy would lower α weights of deeper classiﬁers to a very small value (due to poor initial performance as…”
Multi-level Residual Networks from Dynamical Systems View (2017, background): “Mollifying networks (Gulcehre et al., 2016) start the optimization with an easier (possibly convex) objective function and let it evolve during the training, until it eventually goes back to being the original, difﬁcult to optimize, objective function.”
Regularization for Deep Learning: A Taxonomy (2017, background): “Mollifying networks (Gülçehre et al., 2016b) Connections between layers; noise The mapping can be easier approximated by estimating its decreasingly linear simplified version”
Min-Entropy Latent Model for Weakly Supervised Object Detection (2018, background): “The non-convex optimization problem is turned into a proximate problem, which is easier to be optimized [49], [50].”
Kickstarting Deep Reinforcement Learning (2018, methodology): “Then through adaptation of λ k during the course of learning, the agent is able to shift its optimization focus on the (potentially sparse) reward signal r t , similar to how continuation methods relax optimisation problems to make ﬁnding the solution easier (Gulcehre et al., 2016; Mobahi, 2016).”