The intriguing role of module criticality in the generalization of deep networks
By Niladri S. Chatterji, Behnam Neyshabur, Hanie Sedghi. 2019
60 citations recorded by the source corpus.
We study the phenomenon that some modules of deep neural networks (DNNs) are more \emph{critical} than others. Meaning that rewinding their parameter values back to initialization, while keeping other modules fixed at the trained parameters, results in a large drop in the network's performance. Our analysis reveals interesting properties of the loss landscape which leads us to propose a complexity measure, called {\em module criticality}, based on the shape of the valleys that connects the initial and final values of the module parameters. We formulate how generalization relates to the module criticality, and show that this measure is able to explain the superior generalization performance of some architectures over others, whereas earlier measures fail to do so.
Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Optimization & Training, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Optimization & Training
Citations in the corpus
- ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks (2021, background): “Moreover, from Theorem 3.2 in Chatterji et al. (2019), the following generalization bound holds under the perturbation: Therefore, the left hand side of the statement can be bounded as where the second inequality follows from Lemma 1 and T − 1 w 2 ≤ 1 η .”