Mathematics of Deep Learning

By René Vidal, Joan Bruna, Raja Giryes, Stefano Soatto. 2017

122 citations recorded by the source corpus.

Recently there has been a dramatic increase in the performance of recognition systems due to the introduction of deep architectures for representation learning and classification. However, the mathematical reasons for this success remain elusive. This tutorial will review recent work that aims to provide a mathematical justification for several properties of deep networks, such as global optimality, geometric stability, and invariance of the learned representations.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Stable and Fair Classification (2019, background): “Moreover, several recent works studied stability in deep neural networks [65, 72].”
Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced (2018, background): “…2016, Choromanska et al., 2015, Du and Lee, 2018, Hardt and Ma, 2016, Bartlett et al., 2018, Haeﬀele and Vidal, 2015, Freeman and Bruna, 2016, Vidal et al., 2017, Safran and Shamir, 2016, Zhou and Feng, 2017, Nguyen and Hein, 2017a,b, Zhou and Feng, 2017, Safran and Shamir, 2017], showing…”
Deep Neural Network Concepts for Background Subtraction: A Systematic Review and Comparative Evaluation (2018, background): “eep nonlinearnetworks? – Generalization: How well do DNNs generalize? How should DNNs be regularized? How to preventunder and over ﬁtting? Both architecture and optimization can impact generalization [145,194,195,219]. Furthermore, several architectures are easier to optimize thanothers [194,195]. F”
A Survey on Neural Network Interpretability (2020, background): “For example, the empirical success of DNNs raises many unsolved questions to theoreticians [30].”
Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation (2019, background): “However, explanations for why stochastic gradient descent (SGD) on big neural nets works so well remain elusive, although significant steps are being taken (Vidal et al., 2017; Gunasekar et al., 2017).”
Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex (2018, background): “While many efforts have been devoted to studying the local minima or saddle points of deep neural networks [42, 68, 55, 36, 62, 61], little is known about the duality gap of deep networks.”
Model-Based Learning for Accelerated, Limited-View 3-D Photoacoustic Tomography (2017, background): “We note, that this could be speed up by initialisation of θk with θk−1 or by more advanced optimisation strategies, see for instance the review [50].”
Analysis of Invariance and Robustness via Invertibility of ReLU-Networks (2018, background): “While there has been a growing effort to rigorously study the mathematics behind deep neural networks (DNNs), see [Vidal et al., 2017] for an overview, their inner workings are only sparsely understood so far.”
Stochastic Conditional Generative Networks with Basis Decomposition (2019, background): “Works on studying the relationship between local and global minima of deep networks [9, 26] also suggest the non-uniqueness of optimal parameters of a deep network.”