Information-Theoretic Local Minima Characterization and Regularization
By Zhiwei Jia, Hao Su. 2019
22 citations recorded by the source corpus.
Recent advances in deep learning theory have evoked the study of generalizability across different local minima of deep neural networks (DNNs). While current work focused on either discovering properties of good local minima or developing regularization techniques to induce good local minima, no approach exists that can tackle both problems. We achieve these two goals successfully in a unified manner. Specifically, based on the observed Fisher information we propose a metric both strongly indicative of generalizability of local minima and effectively applied as a practical regularizer. We provide theoretical analysis including a generalization bound and empirically demonstrate the success of our approach in both capturing and improving the generalizability of DNNs. Experiments are performed on CIFAR-10, CIFAR-100 and ImageNet for various network architectures.
Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Christa Zoufal
- Stefan Woerner
- Wei He
- Zhongzhan Huang
- Aurélien Lucchi
- Senwei Liang
- David Sutter
- Ming Liang
In Computational Mechanics
In Computer Vision and Pattern Recognition
- Wei He
- Zhongzhan Huang
- Senwei Liang
- Ming Liang
- Aurélien Lucchi
- Stefan Woerner
- Christa Zoufal
- Xiaogang Wang
Citations in the corpus
- AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, background): “One of the solutions for mitigating the distribution variation problem is to seek the flat-minimal region in the loss landscape [22, 11, 24, 21], as illustrated in Figure 2.”
- The power of quantum neural networks (2020, methodology): “11 To verify the effective dimension’s ability to capture generalisation behaviour, we conduct a numerical analysis similar to work presented in [48].”