Topology and Geometry of Deep Rectified Network Optimization Landscapes

By C. Freeman, Joan Bruna. 2016

16 citations recorded by the source corpus.

Subfields: Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “Numerical and theoretical studies [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17] show that in the overparametrized regime, the loss landscape of DNNs is not rough with isolated minima as initially thought [18, 19], but instead has connected level sets and presents many flat directions, even near its g”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “[20] C Daniel Freeman and Joan Bruna, “Topology and geom-”
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, background): “This question have been explored in a recent work: in Freeman and Bruna [2016] it shown that for one hidden layer rectified neural networks the solution space is connected which is consistent with the flatness of the landscape.”
Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond (2016, background): “Ngiam et al. (2011) has an in depth review of some of the proposals for approximating the Hessian of the loss function.”
Exponentially vanishing sub-optimal local minima in multilayer neural networks (2017, background): “Another approach would be to bound the size of these basins of attraction, by showing that sufﬁciently large of number of differentiable regions near the DLM are also van-ishingly small (other methods might also help here (Freeman & Bruna, 2016)).”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, methodology): “This includes: work on large-batch learning and the generalization gap [148, 72, 64, 57, 67, 131, 66, 146, 94, 151, 152]; work on Energy Landscape approaches to NN training [70, 149, 44, 123, 30, 29, 27, 65, 47, 12, 104, 150, 50, 83, 82, 95]; work on using weight matrices or properties of weight mat”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “Yet, several numerical and rigorous works [16, 43, 20, 40, 9] suggest a different landscape geometry where the loss function is characterized by a connected level set.”
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias (2019, methodology): “The characterization of the loss landscape along paths connecting different models have been studied recently, e.g. in Freeman and Bruna [2016], Garipov et al. [2018], and Draxler et al. [2018].”