The CLEAR Benchmark: Continual LEArning on Real-World Imagery

By Zhiqiu Lin, Jia Shi, Deepak Pathak, Deva Ramanan. 2022

126 citations recorded by the source corpus.

Continual learning (CL) is widely regarded as crucial challenge for lifelong AI. However, existing CL benchmarks, e.g. Permuted-MNIST and Split-CIFAR, make use of artificial temporal variation and do not align with or generalize to the real-world. In this paper, we introduce CLEAR, the first continual image classification benchmark dataset with a natural temporal evolution of visual concepts in the real world that spans a decade (2004-2014). We build CLEAR from existing large-scale image collections (YFCC100M) through a novel and scalable low-cost approach to visio-linguistic dataset curation. Our pipeline makes use of pretrained vision-language models (e.g. CLIP) to interactively build labeled datasets, which are further validated with crowd-sourcing to remove errors and even inappropriate images (hidden in original YFCC100M). The major strength of CLEAR over prior CL benchmarks is the smooth temporal evolution of visual concepts with real-world imagery, including both high-quality labeled data along with abundant unlabeled samples per time period for continual semi-supervised learning. We find that a simple unsupervised pre-training step can already boost state-of-the-art CL algorithms that only utilize fully-supervised data. Our analysis also reveals that mainstream CL evaluation protocols that train and test on iid data artificially inflate performance of CL system. To address this, we propose novel"streaming"protocols for CL that always test on the (near) future. Interestingly, streaming protocols (a) can simplify dataset curation since today's testset can be repurposed for tomorrow's trainset and (b) can produce more generalizable models with more accurate estimates of performance since all labeled data from each time-period is used for both training and testing (unlike classic iid train-test splits).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “For now, we will start with a typical categorization of existing deep continual works into the three categories of regularization, rehearsal and architectural approaches, in consistency ∙ LWF (LiandHoiem,2016) ∙ EBLL (Rannenetal.,2017) ∙ Pseudorehearsal (Robins,1995) ∙ Pseudo-recurrentnets (French,1”
CGLB: Benchmark Tasks for Continual Graph Learning (2022, background): “However, unlike continual learning on Euclidean data ( e.g. , images, texts, etc.) that has established benchmarks and unified experimental settings [30, 36, 45, 61, 20, 37, 28, 15], CGL is still immature and with very rare public benchmark tasks being available.”
Computationally Budgeted Continual Learning: What Does Matter? (2023, methodology): “, CLEAR [32], Yearbook [58] and FMoW [58], NEVIS22 [9], Continual YFCC100M [12] and Continual Google Landmarks V2 [42].”
TiC-CLIP: Continual Training of CLIP Models (2023, unknown): “Traditionally, this field concentrated on synthetic incremental benchmarks that lack natural evolution between tasks, and hence, continual learning methods are seldom used in real-world scenarios (Cossu et al., 2022; Lin et al., 2021).”
Learning from One Continuous Video Stream (2023, background): “Multiple benchmarks have been proposed to measure backward transfer (information retention) and forward transfer (how fast can the model learn a new task, or how generalisable are the features learnt in early tasks), such as CLEAR [23].”
CLNeRF: Continual Learning Meets NeRF (2023, background): “However, this does not model the real-world distribution shifts introduced by the change of time [4, 16], such as the change of scene appearance ( e.g ., lighting and weather) and geometry ( e.g ., new decoration of a room).”