Psycholinguistics Meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering

By Claudio Greco, Barbara Plank, R. Fern'andez, Raffaella Bernardi. 2019

58 citations recorded by the source corpus.

We study the issue of catastrophic forgetting in the context of neural multimodal approaches to Visual Question Answering (VQA). Motivated by evidence from psycholinguistics, we devise a set of linguistically-informed VQA tasks, which differ by the types of questions involved (Wh-questions and polar questions). We test what impact task difficulty has on continual learning, and whether the order in which a child acquires question types facilitates computational models. Our results show that dramatic forgetting is at play and that task difficulty and order matter. Two well-known current continual learning methods mitigate the problem only to a limiting degree.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bing Liu
2. Zixuan Ke
3. Hu Xu
4. Bin Liu
5. Lei Shu
6. Gyuhak Kim
7. Tatsuya Konishi
8. Jun Zhu

In Computer Vision and Pattern Recognition

1. Zixuan Ke
2. Bing Liu
3. Bin Liu
4. Hu Xu
5. Gyuhak Kim
6. Martin Wistuba
7. B. Ermiş
8. Giovanni Zappella

In Question Answering & Reading Comprehension

1. Zixuan Ke
2. Hu Xu
3. Bing Liu
4. Tyler L. Hayes
5. Anette Frank
6. Christopher Kanan
7. Kushal Kafle
8. Manoj Acharya

In Meta & Continual Learning

1. Zixuan Ke
2. Bing Liu
3. Bin Liu
4. Gyuhak Kim
5. Hu Xu
6. Liyuan Wang
7. Lei Shu
8. Xingxing Zhang

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Some other work considers multimodality of vision and language for continual pre-training [81], [467] or downstream tasks [88], [140], [392].”
• Continual Training of Language Models for Few-Shot Learning (2022, background): “…sentence embedding (Liu et al., 2019a), translation (Khayrallah et al., 2018), cross-lingual modeling (Liu et al., 2020b), question answering (Greco et al., 2019) and text classification (Ke et al., 2021a,b; Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020; Mehta et al., 2021;…”
• CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks (2022, background): “For example, one can divide tasks along input categories [Greco et al., 2019] or output classes [Vinyals et al.”
• Memory Efficient Continual Learning with Transformers (2022, background): “BERT [12], have shown their effectiveness in various natural language processing (NLP) tasks such as classification [25], Natural Language Inference [41, 39], and Question Answering [16].”
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, background): “For example, CL models have been proposed for sentiment analysis [23, 24, 37, 43], dialogue slot filling [53], language modeling [58, 7], language learning [31], sentence embedding [33], machine translation [25], cross-lingual modeling [35], and question answering [12].”
• Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[220] further pointed out that besides the color of the objects, question difficulty and order also affect the degree of catastrophic forgetting in the area of VQA, and directly adapting methods from image classification can only mitigate catastrophic forgetting to a small extent.”
• Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, positive): “In Hayes et al. (2020) and Greco, Plank, Fernández, and Bernardi (2019), replay is used as an effective mechanism to mitigate forgetting for the problem of visual question answering, where an agent must answer natural language questions about images.”
• Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “In particular, GLUE (Wang et al., 2018; Greco et al., 2019) and SUPERGLUE (Wang et al., 2019a) benchmarks track performance on eleven and ten language understanding tasks respectively, using existing NLP datasets.”
• Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks (2020, methodology): “, 2019) as well as in V&L tasks in particular (Greco et al., 2019): LXMERT is finetuned on visual question answering in the last 10”
• O N M EMORY IN H UMAN AND A RTIFICIAL L ANGUAGE P ROCESSING S YSTEMS (2020, background): “Catastrophic forgetting—where a model performs poorly on examples from a dataset that is encountered in a distant past (after seeing multiple other datasets)—is a common problem in AI systems (Yogatama et al., 2019; Greco et al., 2019).”
• Continual Pre-training of Language Models (2023, methodology): “In NLP, CL has been used for slot filling (Shen et al., 2019), language learning (Li et al., 2019), sentiment analysis (Ke et al., 2021a), topic modeling (Gupta et al., 2020), question answering (Greco et al., 2019) and text classification (Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020).”