Failures of Gradient-Based Deep Learning

By Shai Shalev-Shwartz, Ohad Shamir, Shaked Shammah. 2017

231 citations recorded by the source corpus.

In recent years, Deep Learning has become the go-to solution for a broad range of applications, often outperforming state-of-the-art. However, it is important, for both theoreticians and practitioners, to gain a deeper understanding of the difficulties and limitations associated with common approaches and algorithms. We describe four types of simple problems, for which the gradient-based algorithms commonly used in deep learning either fail or suffer from significant difficulties. We illustrate the failures through practical experiments, and provide theoretical insights explaining their source, and how they might be remedied.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Relational inductive biases, deep learning, and graph networks (2018, background): “Despite deep learning’s successes, however, important critiques (Shalev-Shwartz et al., 2017; Lake et al., 2017; Lake and Baroni, 2018; Marcus, 2018; Pearl, 2018; Yuille and Liu, 2018) have highlighted key challenges it faces in complex language and scene understanding, reasoning about structured da”
How to Start Training: The Effect of Initialization and Architecture (2018, background): “Among the many possible obstacles to training, it is natural to distinguish two kinds: problems that prevent a given neural network from ever achieving better-than-chance performance and problems that have to do with later stages of training, such as escaping flat regions and saddle points [12, 27],”
Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima (2017, methodology): “Recently, a line of research [Tian, 2017, Brutzkus and Globerson, 2017, Li and Yuan, 2017, Soltanolkotabi, 2017, Shalev-Shwartz et al., 2017b] assumed the input distribution is Gaussian and showed that stochastic gradient descent with random or 0 initialization is able to train a neural network f (…”
DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction (2022, background): “This has motivated a variety of studies that evolves from capturing low-order interactions towards high-order ones [7, 11, 13, 18–20, 22, 23, 29, 30, 34, 40, 43, 43, 48, 51].”
Barren plateaus in quantum neural network training landscapes (2018, methodology): “Also, despite concerns about gradient-based methods in classical deep neural networks [32–34], they are successful [24], even if using random initialization [33, 35].”
Adaptive norms for deep learning with regularized Newton methods (2019, background): “However, gradient descent is known to be inadequate to optimize functions that are ill-conditioned (Nesterov, 2013; Shalev-Shwartz et al., 2017) and thus adaptive gradient methods that employ dynamic, coordinate-wise learning rates based on past gradients—including Adagrad (Duchi et al., 2011),…”
Proving the Lottery Ticket Hypothesis: Pruning is All You Need (2020, background): “Second, weight-optimization is largely done with gradient-based algorithms, which have been shown to be suboptimal in various cases (see [30, 31]).”
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks (2019, background): “Meanwhile, it has been shown that DNNs have no generalization ability in fitting the parity function (Shalev-Shwartz et al., 2017; Nye & Saxe, 2018).”
Learning One-hidden-layer ReLU Networks via Gradient Descent (2018, background): “From a theoretical perspective, it is well-known that learning neural networks in general settings is hard in the worst case (Blum and Rivest, 1989; Auer et al., 1996; Livni et al., 2014; Shamir, 2016; Shalev-Shwartz et al., 2017a,b; Zhang et al., 2017; Ge et al., 2017).”
Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex (2018, background): “While many works have witnessed the power of local search algorithms for deep neural networks [16] these algorithms typically converge to a suboptimal solution in the worst cases according to various empirical observations [52, 28].”
The Pitfalls of Simplicity Bias in Neural Networks (2020, methodology): “Though the k-slab features are special cases of linear periodic functions on which gradient-based methods have been shown to fail for large k [64], we note that we use small values of k ∈ {5, 7} which are quickly learned by SGD in practice.”
AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction (2020, background): “A recent work proves the insensitive gradient issue of DNN when the target is a large collection of uncorrelated functions [25, 28].”
Predicting many properties of a quantum system from very few measurements (2020, background): “[66] S.”
SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver (2019, background): “In particular, we show that we can learn the parity function, known to be challenging for deep classifiers (Shalev-Shwartz et al., 2017), with only single bit supervision.”
Optimization Landscape and Expressivity of Deep CNNs (2017, background): “…well known that the optimization problem for training neural networks can have exponentially many local minima (Auer et al., 1996; Safran & Shamir, 2016) and NP-hardness has been shown in many cases (Blum & Rivest., 1989; Sima, 2002; Livni et al., 2014; Shamir, 2017; Shalev-Shwartz et al., 2017).”
Quantum variational algorithms are swamped with traps (2022, unknown): “One might hope then that training Hamiltonian agnostic VQAs is completely analogous to the classical setting, then, and the loss landscape of (4) exhibits the desirable properties that enable train-ability found in classical networks 19,20 .”
A deep learning-based hybrid approach for the solution of multiphysics problems in electrosurgery (2019, background): “Further, the end-to-end learning approach may be slower to converge due to local optima [25].”
VQE method: a short survey and recent developments (2021, unknown): “Similar problems (i.e., gradient-vanishing) existed in the early studies of deep neural networks [87, 88, 89] with mitigation techniques proposed later on.”
When is a Convolutional Filter Easy To Learn? (2017, background): “…point of view, it is well known that training a neural network is hard in the worst cases [Blum and Rivest, 1989, Livni et al., 2014, Šı́ma, 2002, Shalev-Shwartz et al., 2017a,b] and recently, Shamir [2016] showed either “niceness” of the target function or of the input distribution alone is…”