Dropout as an Implicit Gating Mechanism For Continual Learning

By Seyed Iman Mirzadeh, Mehrdad Farajtabar, Hassan Ghasemzadeh. 2020

47 citations recorded by the source corpus.

In recent years, neural networks have demonstrated an outstanding ability to achieve complex learning tasks across various domains. However, they suffer from the "catastrophic forgetting" problem when they face a sequence of learning tasks, where they forget the old ones as they learn new tasks. This problem is also highly related to the "stability-plasticity dilemma". The more plastic the network, the easier it can learn new tasks, but the faster it also forgets previous ones. Conversely, a stable network cannot learn new tasks as fast as a very plastic network. However, it is more reliable to preserve the knowledge it has learned from the previous tasks. Several solutions have been proposed to overcome the forgetting problem by making the neural network parameters more stable, and some of them have mentioned the significance of dropout in continual learning. However, their relationship has not been sufficiently studied yet. In this paper, we investigate this relationship and show that a stable network with dropout learns a gating mechanism such that for different tasks, different paths of the network are active. Our experiments show that the stability achieved by this implicit gating plays a very critical role in leading to performance comparable to or better than other involved continual learning algorithms to overcome catastrophic forgetting.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Dropout [393] in a stable network behaves like a gating mechanism to create task-specific pathways and thus can mitigate catastrophic forgetting [307].”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, background): “This problem is closely related to the stability-plasticity dilemma [31, 32].”
Wide Neural Networks Forget Less Catastrophically (2021, background): “Unfortunately, however, there is not much fundamental understanding of the intrinsic properties of neural networks that affect continual learning performance through catastrophic forgetting or forward/backward transfer (Mirzadeh et al., 2020b).”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “Note that data-agnostic regularization terms, such as drop-out [29], and the training regime in general [30,31] can have similar effects of altering how credit assignment is done.”
Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, background): “• Dropout (Mirzadeh et al., 2020): Dropout Hin-ton et al. (2012) is recently shown by (Mirzadeh et al., 2020) that it effectively alleviates catastrophic forgetting.”
Understanding the Role of Training Regimes in Continual Learning (2020, background): “Some of them, like dropout, had been previously proposed to help continual learning [20, 52].”
The Effectiveness of Memory Replay in Large Scale Continual Learning (2020, methodology): “Expansion based methods (Rusu et al., 2016; Li & Hoiem, 2017; Yoon et al., 2018; Li et al., 2019; Mirzadeh et al., 2020) tackle the problem by expanding the neural architecture along the training.”
TiC-CLIP: Continual Training of CLIP Models (2023, unknown): “Regularization methods push the model to change slowly in the directions of prior knowledge and often incur additional memory/compute costs (Kirkpatrick et al., 2017; Mirzadeh et al., 2020a;b; Farajtabar et al., 2020).”
Task-agnostic Continual Learning with Hybrid Probabilistic Models (2021, unknown): “Modularity-based methods allocate diﬀerent subsets of the parameters to each task [Rusu et al., 2016, Yoon et al., 2018, Jerfel et al., 2019, Li et al., 2019, Wortsman et al., 2020, Mirzadeh et al., 2020a].”
ADER: Adaptively Distilled Exemplar Replay Towards Continual Learning for Session-based Recommendation (2020, background): “• Dropout [29]: Dropout [13] is recently found by [29] that it effectively alleviates catastrophic forgetting.”
Continual Learning, Fast and Slow (2021, result): “Lastly, a recent work [77] also show promising results of applying dropout in continual learning.”
Efficient Continual Learning Ensembles in Neural Network Subspaces (2022, methodology): “We used the same single training epoch per task setting as in (Chaudhry et al., 2019b; Mirzadeh et al., 2020b,a).”