Open Problem: The landscape of the loss surfaces of multilayer networks

By Anna Choromanska, Yann LeCun, Gérard Ben Arous. 2015

128 citations recorded by the source corpus.

Subfields: Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

The Hidden Vulnerability of Distributed Learning in Byzantium (2018, methodology): “…of ﬁnding “the best direction” possible for non–convex cost functions remains one of the most challenging ones in optimization for machine learning (Choromanska et al., 2015b), especially when we stick to such a cheap, ﬁrst– order method as SGD, and avoid expensive, Hessian–like, computations as…”
Entropy-SGD: biasing gradient descent into wide valleys (2016, background): “Indeed, as discussed in numerous places in literature (Bray & Dean, 2007; Dauphin et al., 2014; Choromanska et al., 2015a), SGD finds low-index critical points, i.e., optimizers with few negative eigenvalues of the Hessian.”
Representation Learning on Graphs with Jumping Knowledge Networks (2018, methodology): “With a randomization assumption of the ReLU activations similar to that in (Kawaguchi, 2016; Choromanska et al., 2015), we can draw connections between GCNs and random walks: Theorem 1.”
Convergence Analysis of Two-layer Neural Networks with ReLU Activation (2017, background): “The assumption that the input follows a Gaussian distribution is not necessarily true in practice (Although this is a common assumption appeared in the previous papers [5, 41, 42], and also considered plausible in [6]).”
An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis (2017, background): “For example, (Choromanska et al., 2015a;b) relates the nonlinear ReLU network with spin-glass models when several assumptions hold, including the assumption of independent activations ( A1p and A5u ).”
An empirical analysis of the optimization of deep network loss surfaces (2016, background): “Others [7, 8, 17] characterize the critical points of fully-connected, nonlinear deep networks, but make simplifying assumptions about the distributions and independence of various variables in the network.”
Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima (2017, background): “…although this is a common assumption appeared in the previous papers [Brutzkus and Globerson, 2017, Li and Yuan, 2017, Zhong et al., 2017a,b, Tian, 2017, Choromanska et al., 2015a, Xie et al., 2017, Shalev-Shwartz et al., 2017b] and also considered plausible in [Choromanska et al., 2015b].”
The Loss Surface of Deep and Wide Neural Networks (2017, methodology): “This is a very interesting result but is based on a number of unrealistic assumptions (Choromanska et al., 2015b).”
Asymmetric Valleys: Beyond Sharp and Flat Local Minima (2019, background): “The loss landscape of neural networks has attracted great research interests in the deep learning community [9, 10, 32, 12, 15, 43, 36].”
Optimization Landscape and Expressivity of Deep CNNs (2017, background): “This is a very interesting result but is based on a number of unrealistic assumptions (Choromanska et al., 2015b).”
Connections with Robust PCA and the Role of Emergent Sparsity in Variational Autoencoder Models (2018, background): “This mirrors the rich tradition of analyzing deep networks under various simplifications such as linear layers or i.i.d. random activation patterns (Choromanska et al., 2015a,b; Goodfellow et al., 2016; Kawaguchi, 2016; Saxe et al., 2014), and results in the following key contributions: 1.”
Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex (2018, background): “Kawaguchi [35] improved over the results of [20, 19] by assuming that the activation functions are independent Bernoulli variables and the input data are drawn from Gaussian distribution.”
The loss surface and expressivity of deep convolutional neural networks (2017, background): “By randomizing the nonlinear part of a feedforward network with ReLU activation function and making some additional simplifying assumptions, Choromanska et al. (2015a) can relate the loss surface of neural networks to a certain spin glass model.”
Hidden Talents of the Variational Autoencoder. (2017, background): “random activation patterns [9, 10, 16, 20, 37], and results in the following key contributions:”
Theory II: Landscape of the Empirical Risk in Deep Learning (2017, methodology): “Techniques borrowed from the physics of spin glasses (which in turn were based on old work by Marc Kac on the zeros of algebraic equations) were used [5] to suggest the existence of a band of local minima of high quality as measured by the test error.”
Graph Neural Networks Exponentially Lose Expressive Power for Node Classification (2019, methodology): “To handle the non-linearity, they linearized networks by a randomization assumption (Choromanska et al., 2015).”