Conditional Computation for Continual Learning

By Min Lin, Jie Fu, Yoshua Bengio. 2019

12 citations recorded by the source corpus.

Catastrophic forgetting of connectionist neural networks is caused by the global sharing of parameters among all training examples. In this study, we analyze parameter sharing under the conditional computation framework where the parameters of a neural network are conditioned on each input example. At one extreme, if each input example uses a disjoint set of parameters, there is no sharing of parameters thus no catastrophic forgetting. At the other extreme, if the parameters are the same for every example, it reduces to the conventional neural network. We then introduce a clipped version of maxout networks which lies in the middle, i.e. parameters are shared partially among examples. Based on the parameter sharing analysis, we can locate a limited set of examples that are interfered when learning a new example. We propose to perform rehearsal on this set to prevent forgetting, which is termed as conditional rehearsal. Finally, we demonstrate the effectiveness of the proposed method in an online non-stationary setup, where updates are made after each new example and the distribution of the received example shifts over time.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

1. Quoc V. Le
2. Andrew M. Dai
3. Wei Wei
4. Tameem Adel
5. Richard E. Turner
6. Min Sun
7. Nan Du
8. Yanping Huang

Citations in the corpus

• Mixture-of-Experts with Expert Choice Routing (2022, background): “Conditional Computation: Computation decisions can be made dynamically based on the input [23, 25].”
• Continual Learning with Adaptive Weights (CLAW) (2019, background): “A clipped version of maxout networks (Goodfellow et al., 2013) is developed in (Lin et al., 2018) where parameters are partially shared among examples.”
• Mitigating Forgetting in Online Continual Learning via Instance-Aware Parameterization (2020, methodology): “Several conditional computation methods (e.g., using node sharpening [24], maxout [25], compete to compute [26]) have been proposed to mitigate catastrophic forgetting.”