Does an LSTM forget more than a CNN? An empirical study of catastrophic forgetting in NLP

By Afshin Rahimi, Timothy Baldwin. 2019

30 citations recorded by the source corpus.

Subfields: Natural Language Processing, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Ting-Wei Liu
2. Wanxiang Che
3. Yiming Cui
4. Minlie Huang
5. Lidong Bing
6. Liying Cheng
7. Lizhen Qu
8. Scott Sanner

In Meta & Continual Learning

1. Zheda Mai
2. Scott Sanner
3. Hyunwoo J. Kim
4. Jihwan Jeong
5. Eugene Belilovsky
6. Ruiwen Li
7. Fei Mi
8. David Quispe

Citations in the corpus

• Probing Representation Forgetting in Supervised and Unsupervised Continual Learning (2022, background): “Recent works on elucidating the nature of catastrophic forgetting have examined the influence of task sequence [33], network architecture [3], and change in representation similarity [38].”
• Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers (2021, positive): “The same phenomenon is also observed in (Arora et al., 2019) that the models with pre-trained language models obtain inferior performance than LSTM or CNN, when fine-tuning incrementally on each task.”
• Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Recent works have confirmed that RNNs, including LSTMs, are also immensely affected by CF [132, 133, 134].”
• Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, background): “Yogatama et al. (2019); Arora et al. (2019) identified the detrimental catastrophic forgetting issue while fine-tuning ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019).”
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics (2020, background): “Otherprior work on understanding catastrophic forgetting has looked at properties of task sequences [33] or as a means of comparing different neural network architectures [34].”
• On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021, background): “This phenomenon often appears in NLP tasks (Mou et al., 2016; Arora et al., 2019).”
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020, background): “Catastrophic forgetting in NLP has raised increased attention recently (Mou et al., 2016; Arora et al., 2019; Chronopoulou et al., 2019).”