RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning

By Riccardo Del Chiaro, Bartłomiej Twardowski, Andrew D. Bagdanov, Joost van de Weijer. 2020

54 citations recorded by the source corpus.

Research on continual learning has led to a variety of approaches to mitigating catastrophic forgetting in feed-forward classification networks. Until now surprisingly little attention has been focused on continual learning of recurrent models applied to problems like image captioning. In this paper we take a systematic look at continual learning of LSTM-based models for image captioning. We propose an attention-based approach that explicitly accommodates the transient nature of vocabularies in continual image captioning tasks -- i.e. that task vocabularies are not disjoint. We call our method Recurrent Attention to Transient Tasks (RATT), and also show how to adapt continual learning approaches based on weight egularization and knowledge distillation to recurrent continual learning problems. We apply our approaches to incremental image captioning problem on two new continual learning benchmarks we define using the MS-COCO and Flickr30 datasets. Our results demonstrate that RATT is able to sequentially learn five captioning tasks while incurring no forgetting of previously learned ones.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Infectious Diseases, Natural Language Processing, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Joost van de Weijer
2. Bartłomiej Twardowski
3. Andrew D. Bagdanov
4. Jun Zhu
5. Hang Su
6. Marc Masana
7. Liyuan Wang
8. Giuseppe Fiameni

In Computer Vision and Pattern Recognition

1. Joost van de Weijer
2. Andrew D. Bagdanov
3. Bartłomiej Twardowski
4. Rita Cucchiara
5. Jun Zhu
6. Marcella Cornia
7. Lorenzo Baraldi
8. Hang Su

In Infectious Diseases

1. Xialei Liu
2. Simone Scardapane
3. Valerio Marsocci

In Natural Language Processing

1. Jorge Armando Mendez Mendez
2. Liyuan Wang
3. Eric Eaton
4. Jun Zhu
5. Rita Cucchiara
6. Andrew D. Bagdanov
7. Lorenzo Baraldi
8. Matteo Stefanini

In Meta & Continual Learning

1. Joost van de Weijer
2. Bartłomiej Twardowski
3. Andrew D. Bagdanov
4. Liyuan Wang
5. Jorge Armando Mendez Mendez
6. Marc Masana
7. Xingxing Zhang
8. Xialei Liu

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Some other work considers multimodality of vision and language for continual pre-training [81], [467] or downstream tasks [88], [140], [392].”
• Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, background): “[99] study preventing forgetting for the task of”
• Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization (2022, background): “This can also be seen in several reviews [4,5,11,29,30,35] reporting few approaches that consider architectures besides CNNs, despite the attempts to investigate CL in RNNs [9,43].”
• From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “with different vocabularies, as proposed in [191], and the model should be able to transfer visual concepts from one to the other while enlarging its vocabulary.”
• How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “The approaches above, originally applied to vision domains, have been extended to image captioning, demonstrating their applicability to language domains (Del Chiaro et al., 2020).”