Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting

By Zeke Xie, Fengxiang He, Shaopeng Fu, Issei Sato, Dacheng Tao, Masashi Sugiyama. 2020

71 citations recorded by the source corpus.

Deep learning is often criticized by two serious issues that rarely exist in natural nervous systems: overfitting and catastrophic forgetting. It can even memorize randomly labeled data, which has little knowledge behind the instance-label pairs. When a deep network continually learns over time by accommodating new tasks, it usually quickly overwrites the knowledge learned from previous tasks. Referred to as the neural variability, it is well known in neuroscience that human brain reactions exhibit substantial variability even in response to the same stimulus. This mechanism balances accuracy and plasticity/flexibility in the motor learning of natural nervous systems. Thus, it motivates us to design a similar mechanism, named artificial neural variability (ANV), that helps artificial neural networks learn some advantages from “natural” neural networks. We rigorously prove that ANV plays as an implicit regularizer of the mutual information between the training data and the learned model. This result theoretically guarantees ANV a strictly improved generalizability, robustness to label noise, and robustness to catastrophic forgetting. We then devise a neural variable risk minimization (NVRM) framework and neural variable optimizers to achieve ANV for conventional network architectures in practice. The empirical studies demonstrate that NVRM can effectively relieve overfitting, label noise memorization, and catastrophic forgetting at negligible costs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wei Zhou
2. Chengyu Wang
3. Yake Wei
4. Jingwei Zhao
5. P. Heng
6. Andong Deng
7. Jun Huang
8. Xiaokang Peng

In Computer Vision and Pattern Recognition

1. Wei Zhou
2. Yake Wei
3. Jingwei Zhao
4. P. Heng
5. Andong Deng
6. Xiaokang Peng
7. Di Hu
8. Pengfei Chen

Citations in the corpus

• MeLL: Large-scale Extensible User Intent Classification for Dialogue Systems with Meta Lifelong Learning (2021, background): “In lifelong learning, it is challenging to address the catastrophic forgetting problem [46] where the model “forgets” how to solve existing tasks when it learns new tasks.”
• Noise against noise: stochastic label noise helps combat inherent label noise (2021, methodology): “The finding motivates several robust learning methods (Harutyunyan et al., 2020; Xie et al., 2020) and also supports our method.”
• Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, background): “The gradient noise of SGD is considered to have an essential correlation with the generalization ability of the deep models [6, 13, 42, 44, 48].”