Doing the Impossible: Why Neural Networks Can Be Trained at All

By Nathan Oken Hodas, Panos Stinis. 2018

21 citations recorded by the source corpus.

As deep neural networks grow in size, from thousands to millions to billions of weights, the performance of those networks becomes limited by our ability to accurately train them. A common naive question arises: if we have a system with billions of degrees of freedom, don't we also need billions of samples to train it? Of course, the success of deep learning indicates that reliable models can be learned with reasonable amounts of data. Similar questions arise in protein folding, spin glasses and biological neural networks. With effectively infinite potential folding/spin/wiring configurations, how does the system find the precise arrangement that leads to useful and robust results? Simple sampling of the possible configurations until an optimal one is reached is not a viable option even if one waited for the age of the universe. On the contrary, there appears to be a mechanism in the above phenomena that forces them to achieve configurations that live on a low-dimensional manifold, avoiding the curse of dimensionality. In the current work we use the concept of mutual information between successive layers of a deep neural network to elucidate this mechanism and suggest possible ways of exploiting it to accelerate training. We show that adding structure to the neural network leads to higher mutual information between layers. High mutual information between layers implies that the effective number of free parameters is exponentially smaller than the raw number of tunable weights, providing insight into why neural networks with far more weights than training points can be reliably trained.

Subfields: Artificial Intelligence, Materials Chemistry

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xuhui Meng
2. Dongkun Zhang
3. G. Em Karniadakis
4. Zhen Li
5. Maizar Raissi
6. Paris Perdikaris
7. Liu Yang
8. Lu Lu

In Materials Chemistry

1. Xuhui Meng
2. G. Em Karniadakis
3. Dongkun Zhang
4. Zhen Li

Citations in the corpus

• PPINN: Parareal Physics-Informed Neural Network for time-dependent PDEs (2019, methodology): “For example, given a DNN with architecture of [2] + [20] ∗ 3 + [1], the wall time of training the PINN (1D Burgers equation) for 10,000 steps is about 100 seconds on an Intel Xeon CPU (E5-2643) and 172 seconds on a NVIDIA GPU (GTX Titan XP) for a training data set consisting of 5,000 points.”