Deep relaxation: partial differential equations for optimizing deep neural networks

By Pratik Chaudhari, Adam M. Oberman, Stanley J. Osher, Stefano Soatto, Guillaume Carlier. 2017

164 citations recorded by the source corpus.

Entropy-SGD is a first-order optimization method which has been used successfully to train deep neural networks. This algorithm, which was motivated by statistical physics, is now interpreted as gradient descent on a modified loss function. The modified, or relaxed, loss function is the solution of a viscous Hamilton–Jacobi partial differential equation (PDE). Experimental results on modern, high-dimensional neural networks demonstrate that the algorithm converges faster than the benchmark stochastic gradient descent (SGD). Well-established PDE regularity results allow us to analyze the geometry of the relaxed energy landscape, confirming empirical evidence. Stochastic homogenization theory allows us to better understand the convergence of the algorithm. A stochastic control interpretation is used to prove that a modified algorithm converges faster than SGD in expectation.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Statistics and Probability

Citations in the corpus

Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks (2017, background): “This idea has also been successful in deep learning where Chaudhari et al. (2016) modified SGD to seek solutions in “wide minima” with low curvature to obtain improvements in generalization performance as well as convergence rate (Chaudhari et al., 2017a).”
A regularization interpretation of the proximal point method for weakly convex functions (2020, methodology): “In connection with the regularization interpretation of [12], we study the convex analytical properties of the regularized function corresponding the proximal point method, in the weakly convex case.”
A unified deep artificial neural network approach to partial differential equations in complex geometries (2017, methodology): “In the following we let B in (3.1) be the identity operator and we hence consider PDEs with Dirichlet boundary conditions.”
Mathematics of Deep Learning (2017, background): “Surprisingly, these techniques from statistical physics are intimately connected to the regularization properties of partial differential equations (PDEs) [26].”
Deep Neural Networks Motivated by Partial Differential Equations (2018, unknown): “Similar observations were made in [10,49].”
On Learning Rates and Schrödinger Operators (2020, methodology): “In the stochastic setting, this approach has been recently pursued by various authors [COO + 18, CS18, MHB16, LSJR16, CH19, LTE17] to establish various properties of stochastic optimization.”