On the Learning Dynamics of Deep Neural Networks

By Rémi Tachet des Combes, Mohammad Pezeshki, Samira Shabanian, Aaron C. Courville, Yoshua Bengio. 2018

44 citations recorded by the source corpus.

While a lot of progress has been made in recent years, the dynamics of learning in deep nonlinear neural networks remain to this day largely misunderstood. In this work, we study the case of binary classification and prove various properties of learning in such networks under strong assumptions such as linear separability of the data. Extending existing results from the linear case, we confirm empirical observations by proving that the classification error also follows a sigmoidal shape in nonlinear architectures. We show that given proper initialization, learning expounds parallel independent modes and that certain regions of parameter space might lead to failed training. We also demonstrate that input norm and features' frequency in the dataset lead to distinct convergence speeds which might shed some light on the generalization capabilities of deep neural networks. We provide a comparison between the dynamics of learning with cross-entropy and hinge losses, which could prove useful to understand recent progress in the training of generative adversarial networks. Finally, we identify a phenomenon that we baptize \textit{gradient starvation} where the most frequent features in a dataset prevent the learning of other less frequent but equally informative features.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “Direct minimization of this loss function using the gradient descent obeys coupled dynamics and is difficult to treat directly [26].”
Edinburgh Research Explorer Understanding Learning Dynamics Of Language Models with SVCCA (unknown): “Combes et al. (2018) identiﬁed the phenomenon of gradient starvation , meaning that while frequent and unambiguous features are learned quickly in training, they slow down the learning of rarer features.”