Do Image Classifiers Generalize Across Time?

By Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, B. Recht, Ludwig Schmidt. 2019

87 citations recorded by the source corpus.

Vision models notoriously flicker when applied to videos: they correctly recognize objects in some frames, but fail on perceptually similar, nearby frames. In this work, we systematically analyze the robustness of image classifiers to such temporal perturbations in videos. To do so, we construct two new datasets, ImageNet-Vid-Robust and YTBB-Robust, containing a total of 57,897 images grouped into 3,139 sets of perceptually similar images. Our datasets were derived from ImageNet-Vid and Youtube-BB, respectively, and thoroughly re-annotated by human experts for image similarity. We evaluate a diverse array of classifiers pre-trained on ImageNet and show a median classification accuracy drop of 16 and 10 points, respectively, on our two datasets. Additionally, we evaluate three detection models and show that natural perturbations induce both classification as well as localization errors, leading to a median drop in detection mAP of 14 points. Our analysis demonstrates that perturbations occurring naturally in videos pose a substantial and realistic challenge to deploying convolutional neural networks in environments that require both reliable and low-latency predictions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “The noisy ﬁt on YTBB-Robust is likely due to the fact that the categories in YTBB-Robust are not well aligned with those of ImageNet, where the models were trained [76].”
Toward Causal Representation Learning (2021, background): “…and detection methods with respect to simple algorithmically generated interventions, such as spatial shifts, blur, changes in brightness or contrast [107], [170], time consistency [95], [227], control over background and rotation [12], as well as images collected in multiple environments [20].”
Towards Causal Representation Learning (2021, background): “were proposed to specifically test generalization of classification and detection methods with respect to simple algorithmically generated interventions like spatial shifts, blur, changes in brightness or contrast [106, 170], time consistency [94, 227], control over background and rotation [11], as ”
Robust fine-tuning of zero-shot models (2021, background): “Figure 8: WiSE-FT improves OOD accuracy while maintaining or improving ID accuracy on ImageNet-Vid-Robust, YTBB-Robust (Shankar et al., 2019), CIFAR-10.”
Data-centric Artificial Intelligence: A Survey (2023, unknown): “To illustrate, some studies [85, 201] generate various sets of contiguous video frames that appear visually similar to humans but lead to inconsistent predictions due to the small perturbations.”
Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization (2021, methodology): “While Taori et al. (2020) later showed that linear trends still occur for Ima-geNet models on datasets like ObjectNet, Vid-Robust, and YTBB-Robust (Barbu et al., 2019; Shankar et al., 2019), all of their experiments were limited to ImageNet-like tasks.”
Deep Ensembles for Low-Data Transfer Learning (2020, background): “– YouTube Bounding Boxes (YTBB - (Real et al., 2017; Shankar et al., 2019) – ImageNet Vidrobust (Deng et al., 2009; Shankar et al., 2019)”
Partial success in closing the gap between human and machine vision (2021, background): “Many OOD generalisation tests have been proposed: ImageNet-C [15] for corrupted imaes, ImageNet-Sketch [16] for sketches, Stylized-ImageNet [17] for image style changes, [18] for unfamiliar object poses, and many more [19–29].”
Test-Time Training with Self-Supervision for Generalization under Distribution Shifts (2019, methodology): “These include images with diverse types of corruption at various levels (Hendrycks & Dietterich, 2019), video frames of moving objects (Shankar et al., 2019), and a new test set of unknown shifts collected by (Recht et al.”