The loss surface and expressivity of deep convolutional neural networks

By Quynh N. Nguyen, Matthias Hein. 2017

29 citations recorded by the source corpus.

We analyze the expressiveness and loss surface of practical deep convolutional neural networks (CNNs) with shared weights and max pooling layers. We show that such CNNs produce linearly independent features at a "wide" layer which has more neurons than the number of training samples. This condition holds e.g. for the VGG network. Furthermore, we provide for such wide CNNs necessary and sufficient conditions for global minima with zero training error. For the case where the wide layer is followed by a fully connected layer, we show that almost every critical point of the empirical loss is a global minimum with zero training error. Our analysis suggests that both depth and width are very important in deep learning. While depth brings more representational power and allows the network to learn high level features, width smoothes the optimization landscape of the loss function in the sense that a sufficiently wide network has a well-behaved loss surface with potentially no bad local minima.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Simon Shaolei Du
2. Wei Hu
3. Jason D. Lee
4. Yu Bai
5. Quanquan Gu
6. Yuanzhi Li
7. Yongrui Ma
8. Zhihao Shi

In Computer Vision and Pattern Recognition

1. Simon Shaolei Du
2. Jason D. Lee
3. Yu Bai
4. Wei Hu
5. Yongrui Ma
6. Zhihao Shi
7. Laura Palagi
8. Xiaohong Liu

Citations in the corpus

• Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced (2018, background): “…and Ma, 2016, Bartlett et al., 2018, Haeffele and Vidal, 2015, Freeman and Bruna, 2016, Vidal et al., 2017, Safran and Shamir, 2016, Zhou and Feng, 2017, Nguyen and Hein, 2017a,b, Zhou and Feng, 2017, Safran and Shamir, 2017], showing that the objective functions have properties (ii) and (iii).”
• Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks (2019, background): “…Many prior works have tried to establish favorable landscape properties such as every local minimum is a global minimum (Ge et al., 2017; Du & Lee, 2018; Soltanolkotabi et al., 2018; Hardt & Ma, 2016; Freeman & Bruna, 2016; Nguyen & Hein, 2017a;b; Haeffele & Vidal, 2015; Venturi et al., 2018).”
• GridDehazeNet: Attention-Based Multi-Scale Network for Image Dehazing (2019, background): “More fundamentally, it is known that the loss surface of a CNN is generally well-behaved in the sense that the local minima are often almost as good as the global minimum [3, 4, 25].”
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation (2018, background): “Second, we consider another form of over-parametrization, This condition on the amount of over-parameterization is much milder than k ≥ n , a condition used in many previous papers (Nguyen and Hein, 2017a;b).”
• Global optimization issues in deep network regression: an overview (2018, background): “The same authors extended the analysis to deep convolutional neural networks (CNNs) with standard architecture [101] proving that the training problem basically becomes linearly separable at the wide hidden layer of the CNNs.”