Learning with Long-term Remembering: Following the Lead of Mixed Stochastic Gradient

By Yunhui Guo, Mingrui Liu, Tianbao Yang, Tajana Šimunić. 2019

10 citations recorded by the source corpus.

Current deep neural networks can achieve remarkable performance on a single task. However, when the deep neural network is continually trained on a sequence of tasks, it seems to gradually forget the previous learned knowledge. This phenomenon is referred to as catastrophic forgetting and motivates the field called lifelong learning. The central question in lifelong learning is how to enable deep neural networks to maintain performance on old tasks while learning a new this http URL this paper, we introduce a novel and effective lifelong learning algorithm, calledMixEd stochastic GrAdient (MEGA), which allows deep neural networks to ac-quire the ability of retaining performance on old tasks while learning new tasks.MEGA modulates the balance between old tasks and the new task by integrating the current gradient with the gradient computed on a small reference episodic memory. Extensive experimental results show that the proposed MEGA algorithm significantly advances the state-of-the-art on all four commonly used life-long learning benchmarks, reducing the error by up to 18%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zihan Ye
2. Fan Lyu
3. Song Wang
4. Shuai Wang
5. Fuyuan Hu
6. Wei Feng

In Computer Vision and Pattern Recognition

1. Zihan Ye
2. Fan Lyu
3. Song Wang
4. Shuai Wang
5. Fuyuan Hu
6. Wei Feng

Citations in the corpus

• Exploring Example Influence in Continual Learning (2022, neutral): “To suppress the catastrophic forgetting, the rehearsal-based CL [30, 22, 32, 6, 16] builds a small size memory bufferMt sampled from Dtrn t for each task (i.”
• Multi-Domain Multi-Task Rehearsal for Lifelong Learning (2020, methodology): “We compare the proposed method with the state-of-theart methods including EWC (Kirkpatrick et al. 2017), MAS (Aljundi et al. 2018), RWalk (Chaudhry et al. 2018a), PI (Zenke, Poole, and Ganguli 2017), GEM (Lopez-Paz and Ranzato 2017), MER (Riemer et al. 2018), ER (Chaudhry et al. 2019b) A-GEM (Chaudh”