Global Optimality in Neural Network Training

By Benjamin David Haeffele, René Vidal. 2017

125 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Visualizing the Loss Landscape of Neural Nets (2017, background): “For networks with specific structures, there likely exists a monotonically decreasing path from an initialization to a global minimum [32, 15].”
Overcoming Multi-Model Forgetting (2019, positive): “To this end, we trained A until convergence, thus obtaining a solution close to the MLE ˆ θ A = ( ˆ θ 1 , ˆ θ s ) , since all our operations are positively homogeneous (Haeffele & Vidal, 2017).”
A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks (2018, background): “ure convergence to global minimum. Many works, both classic (e.g. Baldi and Hornik (1989)) and recent (e.g. Choromanska et al. (2015); Kawaguchi (2016); Hardt and Ma (2016); Soudry and Carmon (2016); Haeffele and Vidal (2017); Nguyen and Hein (2017); Safran and Shamir (2018); Nguyen and Hein (2018);”
On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport (2018, background): “• Low-rank tensor decomposition [16], recovering mixture models from sketches [26], see [6] for a detailed list of other applications.”
Mathematics of Deep Learning (2017, background): “The results of [30], [31] also provide strong guidance on the design of network regularization to ensure the non-existence of spurious local minima, showing that traditional weight decay is not appropriate for deep networks.”
Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory (2019, background): “Generalization bounds from Harvey et al. (2017) are improved under conditions of low rank and high sparsity (Neyshabur et al.”
Optimization Landscape and Expressivity of Deep CNNs (2017, unknown): “NO results have been recently developed (Andoni et al., 2014; Sedghi & Anandkumar, 2015; Janzamin et al., 2016; Gautier et al., 2016; Haeffele & Vidal, 2017; Soudry & Hoffer, 2017; Goel & Klivans, 2017; Li & Yuan, 2017; Zhong et al., 2017; Tian, 2017; Brutzkus & Globerson, 2017; Soltanolkotabi,…”
Learning Action Representations for Reinforcement Learning (2019, background): “Multiple works have established that for such models, there are no spurious local minimas and the Hessian at every saddle point has at least one negative eigenvalue [Kawaguchi, 2016, Haeffele and Vidal, 2017, Zhu et al., 2018].”
Width Provably Matters in Optimization for Deep Linear Neural Networks (2019, background): “Many papers tried to identify the two desired geometric landscape properties of objective functions for non-linear neural networks (Freeman & Bruna, 2016; Nguyen & Hein, 2017; Venturi et al., 2018; Soudry & Carmon, 2016; Du & Lee, 2018; Soltanolkotabi et al., 2018; Haeffele & Vidal, 2017).”
Distance geometry and data science (2019, background): “yet unseen input. Currently, nding \good" optima of ANN training problems is mostly based on experience, although a considerable eort is under way in order to reach a sound denition of optimum [58, 197, 81, 47]. The main reason why ANNs are so popular today is that they have proven hugely succe”
Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex (2018, background): “(b) The regularization terms in problem (4) are of common interest, e.g., see [30].”