Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods

By Majid Janzamin, Hanie Sedghi, Anima Anandkumar. 2017

245 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Beyond CCA: Moment Matching for Multi-View Models (2016, background): “…process inference (Tung & Smola, 2014), stochastic languages (Balle et al., 2014), mixture of hidden Markov models (S ¨ ubakan et al., 2014), neural networks (see, e.g., Anandkumar & Sedghi, 2015; Janzamin et al., 2016), and other models (see, e.g., Anandkumar et al., 2014, and references therein).”
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (2018, background): “…certain theoretical guarantees for non-convex optimization, and in particular for training deep neural networks, were recently suggested in various works, for example Ge et al. (2015); Agarwal et al. (2017); Carmon et al. (2016); Janzamin et al. (2015); Livni et al. (2014) and references therein.”
Entropy-SGD: biasing gradient descent into wide valleys (2016, positive): “…empirical observations, e.g., papers such as Choromanska et al. (2015a;b); Chaudhari & Soatto (2015) that build upon results from statistical physics as also others such as Haeffele & Vidal (2015) and Janzamin et al. (2015) that obtain similar results for matrix and tensor factorization problems.”
Convergence Analysis of Two-layer Neural Networks with ReLU Activation (2017, background): “Fortunately, we also show that if g is large, by doing SGD, it will keep decreasing until it is small enough while maintaining the weight W in a nice region.”
No bad local minima: Data independent training error guarantees for multilayer neural networks (2016, methodology): “Also, [17] devised a tensor method, instead of the standard SGD method, for which MNNs with one hidden layer are guaranteed to approximate arbitrary functions.”
An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis (2017, background): “(Janzamin et al., 2015) gives guarantees for parameter recovery of a 2-layered network learnt with tensor decomposition.”
Recovery Guarantees for One-hidden-layer Neural Networks (2017, methodology): “Some existing recovery guarantees for NNs are provided by tensor methods (Sedghi & Anandkumar, 2015; Janzamin et al., 2015).”
Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs (2017, background): “In a diﬀerent approach, (Janzamin et al., 2015), suggest alternatives to gradient-based methods for learning neural networks.”
On the Convergence Rate of Training Recurrent Neural Networks (2018, methodology): “Instead of using randomly initialized weights like this paper, there is a line of work proposing algorithms using weights generated from some “tensor initialization” process [4, 26, 48, 55, 61].”
Learning Deep ResNet Blocks Sequentially using Boosting Theory (2017, methodology): “(Janzamin et al., 2015) propose a tensor decomposition technique which decomposes a tensor formed by some transformation of the features x combined with labels y and recovers the weights of a one-hidden layer neural network with guarantees.”
Gradient Descent with Identity Initialization Efficiently Learns Positive-Definite Linear Transformations by Deep Residual Networks (2018, background): “Other theoretical treatments of neural network learning algorithms include Lee et al. (1996); Arora et al. (2014); Livni et al. (2014); Janzamin et al. (2015); Safran & Shamir (2016); Zhang et al. (2016); Nguyen & Hein (2017); Zhang et al. (2017); Orhan & Pitkow (2018), although these are less…”
Spurious Local Minima are Common in Two-Layer ReLU Neural Networks (2017, background): “Also, several recent works have studied learning neural networks under a Gaussian distribution assumption (e.g., Janzamin et al. [2015], Brutzkus and Globerson [2017], Du et al. [2017], Li and Yuan [2017], Feizi et al. [2017], Zhang et al. [2017], Ge et al. [2017]), but using a network architecture…”
Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima (2017, background): “With some additional assumptions, many works tried to design algorithms that provably learn a neural network with polynomial time and sample complexity [Goel et al., 2016, Zhang et al., 2015, Sedghi and Anandkumar, 2014, Janzamin et al., 2015, Goel and Klivans, 2017a,b].”