Optimization Landscape and Expressivity of Deep CNNs

By Quynh N. Nguyen, Matthias Hein. 2017

118 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

On the Convergence Rate of Training Recurrent Neural Networks (2018, background): “ist certain weights so that the network calculates interesting functions. These results do not usually cover the theoretical issue of how to ﬁnd such weight parameters. We refer interested readers to [15, 45, 51, 53] and the references therein. 2 Notations and Preliminaries Notations. We denote by k”
Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced (2018, background): “…and Ma, 2016, Bartlett et al., 2018, Haeffele and Vidal, 2015, Freeman and Bruna, 2016, Vidal et al., 2017, Safran and Shamir, 2016, Zhou and Feng, 2017, Nguyen and Hein, 2017a,b, Zhou and Feng, 2017, Safran and Shamir, 2017], showing that the objective functions have properties (ii) and (iii).”
Deep learning classifiers for hyperspectral imaging: A review (2019, background): “The training of DNNs is complex, since the optimization and the tuning of parameters in deep models is a non-convex and NP-complete problem (Blum and Rivest, 1989), much harder to train and without guaranteeing the convergence of the optimization process (Chen and Wang, 2014; Nguyen and Hein, 2018).”
The Effect of Network Width on the Performance of Large-batch Training (2018, background): “In addition, [42] shows that larger widths leads to better optimization landscape.”
Information-Theoretic Local Minima Characterization and Regularization (2019, background): “…the optimization process, showing that learning algorithms such as stochastic gradient descent (SGD) tend to end up in one of the many local minima which have close-to-zero training loss (Choromanska et al., 2015; Dauphin et al., 2014; Kawaguchi, 2016; Nguyen & Hein, 2018; Du et al., 2018).”
On the Power of Over-parametrization in Neural Networks with Quadratic Activation (2018, background): “Second, we consider another form of over-parametrization, This condition on the amount of over-parameterization is much milder than k ≥ n , a condition used in many previous papers (Nguyen and Hein, 2017a;b).”
Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks (2019, background): “…prior works have tried to establish favorable landscape properties such as every local minimum is a global minimum (Ge et al., 2017; Du and Lee, 2018; Soltanolkotabi et al., 2018; Hardt and Ma, 2016; Freeman and Bruna, 2016; Nguyen and Hein, 2017a,b; Haeﬀele and Vidal, 2015; Venturi et al., 2018).”
Hierarchically Structured Meta-learning (2019, methodology): “…4 convolutional layers in few-shot image classification (Finn et al., 2017)) with ReLU do not meet the property of Lipschitz Hessian, following (Nguyen & Hein, 2018), a softplus function f(x) = 1 κ log(1 + exp(κx)) can arbitrarily well approximate ReLU by adjusting κ and thus Theorem 2 holds.”
Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks (2019, background): “Hence, there is a multitude of works trying to theoretically explain their remarkable performance; see, e.g., (Du et al., 2018; Nguyen and Hein, 2018; Zhang et al., 2017; Zou et al., 2018).”
Gradient Descent Provably Optimizes Over-parameterized Neural Networks (2018, background): “If m is large enough, one can show the global minimum has zero training loss (Nguyen and Hein, 2018).”
Hyperspectral Image Classification—Traditional to Deep Models: A Survey for Future Prospects (2021, background): “NP-complete problem where the convergence of the optimization process is not guaranteed [93].”
Are All Layers Created Equal? (2019, background): “The capacity to represent arbitrary functions on nite samples is also extensively discussed (Hardt and Ma, 2017; Zhang et al., 2017; Nguyen and Hein, 2018; Yun et al., 2018).”