Data-dependence of plateau phenomenon in learning with neural network—statistical mechanical analysis

By Yuki Yoshida, M. Okada. 2020

42 citations recorded by the source corpus.

The plateau phenomenon, wherein the loss value stops decreasing during the process of learning, has been reported by various researchers. The phenomenon was actively inspected in the 1990s and found to be due to the fundamental hierarchical structure of neural network models. Then, the phenomenon has been thought of as inevitable. However, the phenomenon seldom occurs in the context of recent deep learning. There is a gap between theory and reality. In this paper, using statistical mechanical formulation, we clarified the relationship between the plateau phenomenon and the statistical property of the data learned. It is shown that the data whose covariance has small and dispersed eigenvalues tend to make the plateau phenomenon inconspicuous.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

Citations in the corpus

• Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, neutral): “…al., 2017; Tian, 2017; Du et al., 2018; Soltanolkotabi et al., 2018; Aubin et al., 2018; Saxe et al., 2018; Baity-Jesi et al., 2018; Goldt et al., 2019; Ghorbani et al., 2019; Yoshida & Okada, 2019; Ndirango & Lee, 2019; Gabri ´ e, 2020; Bahri et al., 2020; Zdeborov ´ a, 2020; Advani et al., 2020).”