Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
By Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora, Rong Ge. 2019
106 citations recorded by the source corpus.
Mode connectivity is a surprising phenomenon in the loss landscape of deep nets. Optima -- at least those discovered by gradient-based optimization -- turn out to be connected by simple paths on which the loss function is almost constant. Often, these paths can be chosen to be piece-wise linear, with as few as two segments. We give mathematical explanations for this phenomenon, assuming generic properties (such as dropout stability and noise stability) of well-trained deep nets, which have previously been identified as part of understanding the generalization properties of deep nets. Our explanation holds for realistic multilayer nets, and experiments are presented to verify the theory.
Subfields: Computer Vision and Pattern Recognition, Ecology, Environmental Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Mehrdad Farajtabar
- Seyed Iman Mirzadeh
- Razvan Pascanu
- Hassan Ghasemzadeh
- Dilan Gorur
- Rozhin Nobahari
- Amirreza Shaeiri
- Sheng Xu
In Environmental Engineering
Citations in the corpus
- Linear Mode Connectivity in Multitask and Continual Learning (2020, background): “Recently, various works provided different theoretical explanations for mode connectivity in different scenarios (Venturi et al., 2019; Kuditipudi et al., 2019) either by relying on over-parametrization or concepts such as noise-stability or dropout-stability.”
- Towards Deep Learning Models Resistant to Large Perturbations (2020, background): “…a more theoretical perspective (Attias et al., 2018; Chen et al., 2020; Cullina et al., 2018; Diochnos et al., 2019; Gilmer et al., 2018; Ilyas et al., 2019; Khim & Loh, 2018; Montasser et al., 2019; Schmidt et al., 2018; Wang et al., 2017; Yin et al., 2018; Zhai et al., 2019; Zhang et al., 2019).”
- Editing Models with Task Arithmetic (2022, unknown): “The geometry of neural network loss surfaces has attracted the interest of several authors in recent years [54; 28; 21; 48; 25; 13; 98; 7; 23; 55; 60].”
- Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS (2020, background): “Theoretical results were thereby established regarding the convergence [1, 13, 14], generalization [4, 9], and loss landscape [20] of overparametrized neural networks in the NTK regime.”