CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks

By Tejas Srinivasan, Ting-Yun Chang, Leticia Pinto-Alva, Georgios Chochlakis, Mohammad Rostami, Jesse Thomason. 2022

94 citations recorded by the source corpus.

Current state-of-the-art vision-and-language models are evaluated on tasks either individually or in a multi-task setting, overlooking the challenges of continually learning (CL) tasks as they arrive. Existing CL benchmarks have facilitated research on task adaptation and mitigating"catastrophic forgetting", but are limited to vision-only and language-only tasks. We present CLiMB, a benchmark to study the challenge of learning multimodal tasks in a CL setting, and to systematically evaluate how upstream continual learning can rapidly generalize to new multimodal and unimodal tasks. CLiMB includes implementations of several CL algorithms and a modified Vision-Language Transformer (ViLT) model that can be deployed on both multimodal and unimodal tasks. We find that common CL methods can help mitigate forgetting during multimodal task learning, but do not enable cross-task knowledge transfer. We envision that CLiMB will facilitate research on a new class of CL algorithms for this challenging multimodal setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Some other work considers multimodality of vision and language for continual pre-training [81], [467] or downstream tasks [88], [140], [392].”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “CLiMB (Srinivasan et al., 2022) establishes a benchmark that focuses on cross-modality and will thus be interesting to investigate in going beyond our computer vision examples in this paper.”
Deep Class-Incremental Learning: A Survey (2023, background): “, semantic segmentation [164], [165], [166], [167], [174], object detection [175], [176], [177], [178], [179], [180], [181], video understanding [95], [133], medical surgery [182], [183], language model [184], [185], [186], [187], language-vision model [188], [189], etc.”
TiC-CLIP: Continual Training of CLIP Models (2023, unknown): “While some recent works have started to introduce large-scale continual learning benchmarks, they are not naturally time-continual and are comparatively much smaller in scale (Ni et al., 2023; Srinivasan et al., 2022).”
Class-Incremental Learning: A Survey (2023, background): “…[247], video understanding [91], [161], medical surgery [248], [249], language model [250], [251], [252], [253], language-vision model [217], [254], etc. Specifically, object detection and semantic segmentation are two popular vision tasks that may also face the incremental learning scenario.”
VAuLT: Augmenting the Vision-and-Language Transformer with the Propagation of Deep Language Representations (2022, background): “Moreover, a multimodal model should be able to handle one modality at a time when a post is missing one of the two, as was illustrated by Srinivasan et al. (2022).”