Collapse of Deep and Narrow Neural Nets
By Lu Lu, Yanhui Su, G. Em Karniadakis. 2018
179 citations recorded by the source corpus.
Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem. This problem can be typically resolved by the rectified linear unit (ReLU) activation. However, here we show that even for such activation, deep and narrow neural networks (NNs) will converge to erroneous mean or median states of the target function depending on the loss with high probability. Deep and narrow NNs are encountered in solving partial differential equations with high-order derivatives. We demonstrate this collapse of such NNs both numerically and theoretically, and provide estimates of the probability of collapse. We also construct a diagram of a safe region for designing NNs that avoid the collapse to erroneous states. Finally, we examine different ways of initialization and normalization that may avoid the collapse problem. Asymmetric initializations may reduce the probability of collapse but do not totally eliminate it.
Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- G. Em Karniadakis
- Lu Lu
- Xuhui Meng
- Zhiping Mao
- Guofei Pang
- Shengze Cai
- Zhongqiang Zhang
- Somdatta Goswami
In Cognitive Neuroscience
In Electrical and Electronic Engineering
Citations in the corpus
- fPINNs: Fractional Physics-Informed Neural Networks (2018, methodology): “In this paper, we focus on the NN approaches due to the high expressive power of NNs in function approximation [24, 25, 26, 27].”
- A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data (2021, background): “It has been shown that it could be difficult to train neural networks if the variance of the output of each layer vanishes or explodes [50, 51].”
- DeepXDE: A Deep Learning Library for Solving Differential Equations (2019, background): “However, currently there is no error estimation for PINNs yet, and even quantifying the three errors for supervised learning is still an open research problem [36, 35, 25].”