Energy landscapes for machine learning.

By Andrew J. Ballard, Ritankar Das, Stefano Martiniani, Dhagash Mehta, Levent Sagun, Jacob D. Stevenson, David J. Wales. 2017

101 citations recorded by the source corpus.

Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular potential energy landscapes can be applied to these machine learning landscapes to gain new insight into the solution space involved in training and the nature of the corresponding predictions. In particular, we can define quantities analogous to molecular structure, thermodynamics, and kinetics, and relate these emergent properties to the structure of the underlying landscape. This Perspective aims to describe these analogies with examples from recent applications, and suggest avenues for new interdisciplinary research.

Subfields: Computational Theory and Mathematics, Materials Chemistry, Molecular Biology, Optimization & Training, AI for Physical Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Materials Chemistry

In Molecular Biology

In Optimization & Training

In AI for Physical Sciences

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “Numerical and theoretical studies [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17] show that in the overparametrized regime, the loss landscape of DNNs is not rough with isolated minima as initially thought [18, 19], but instead has connected level sets and presents many flat directions, even near its g”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “[26] Andrew J Ballard, Ritankar Das, Stefano Martiniani, Dhagash Mehta, Levent Sagun, Jacob D Stevenson, and David J Wales, “Energy landscapes for machine learning,” Physical Chemistry Chemical Physics (2017).”
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, background): “A theoretical review on this can be found in [Aufﬁnger et al., 2013], while Sagun et al. [2014] and Ballard et al. [2017] provide an experimental simulation as well as a numerical study for neural networks.”
Essentially No Barriers in Neural Network Energy Landscape (2018, methodology): “In follow-up work, (Ballard et al., 2017) trained a multi-layer perceptron with a single hidden layer on MNIST.”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “Furthermore, studies of the Hessian of the loss function [37, 38, 3] and of the learning ar X iv :1 81 0.”