Improving robustness against common corruptions by covariate shift adaptation

By Steffen Schneider, Evgenia Rusak, L. Eck, Oliver Bringmann, Wieland Brendel, Matthias Bethge. Neural Information Processing Systems, 2020

651 citations recorded by the source corpus.

Today's state-of-the-art machine vision models are vulnerable to image corruptions like blurring or compression artefacts, limiting their performance in many real-world applications. We here argue that popular benchmarks to measure model robustness against common corruptions (like ImageNet-C) underestimate model robustness in many (but not all) application scenarios. The key insight is that in many scenarios, multiple unlabeled examples of the corruptions are available and can be used for unsupervised online adaptation. Replacing the activation statistics estimated by batch normalization on the training set with the statistics of the corrupted images consistently improves the robustness across 25 different popular computer vision models. Using the corrected statistics, ResNet-50 reaches 62.2% mCE on ImageNet-C compared to 76.7% without adaptation. With the more robust AugMix model, we improve the state of the art from 56.5% mCE to 51.0% mCE. Even adapting to a single sample improves robustness for the ResNet-50 and AugMix models, and 32 samples are sufficient to improve the current state of the art for a ResNet-50 architecture. We argue that results with adapted statistics should be included whenever reporting scores in corruption benchmarks and other out-of-distribution generalization settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/aa12f44062c06f90526cb128f10e829846145b1f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Test-Time Domain Adaptation (2022, methodology): “The BN Stats Adapt method keeps the network weights and uses the Batch Normalization statistics from the input data of the current iteration for the prediction [34, 51].”
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift (2020, methodology): “We evaluate the general approach of using test batches to compute batch normalization (BN) statistics, which has been proposed in several prior works [36, 29, 47, 58].”
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models (2022, methodology): “It is a straightforward way to directly adapt the BN statistics at test time to enhance model robustness against distribution shifts [52].”
TTT++: When Does Self-Supervised Test-Time Training Fail or Thrive? (2021, methodology): “One simplest form is to replace the batch-norm statistics estimated on the training set with those on test examples [17].”
MEMO: Test Time Robustness via Adaptation and Augmentation (2021, methodology): “No prior test time adaptation methods have reported improvements on ImageNet-A, and some have reported explicit negative results [32].”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, neutral): “batch normalization calibration PredBN [13], [370], InstCal [371] model optimization TTT [8], GeOS [365], MEMO [14] meta-learning MLSR [372], Full-OSHOT [373] input adaptation TPT [374], TTA-DAE [375] dynamic inference EMEA [376], DRM [377]”
Towards Stable Test-Time Adaptation in Dynamic Wild World (2023, methodology): “In TTA, prior methods often conduct adaptation on pre-trained models with batch normalization (BN) layers (Ioffe & Szegedy, 2015), and most of them are built upon BN statistics adaptation (Schneider et al., 2020; Nado et al., 2020; Khurana et al., 2021; Wang et al., 2021; Niu et al., 2022a; Hu et al”
Efficient Test-Time Model Adaptation without Forgetting (2022, methodology): “…test-time adaptation methods have been proposed to adapt a model with only test data, including batchnorm statistics adaptation (Nado et al., 2020; Schneider et al., 2020; Khurana et al., 2021), test-time entropy minimization (Wang et al., 2021; Fleuret et al., 2021), prediction consistency…”
Inductive biases for deep learning of higher-level cognition (2020, background): “Instead, current systems are generally not robust to changes in distribution (Peters et al., 2017b; Geirhos et al., 2020; Hendrycks et al., 2021; Koh et al., 2021; Schneider et al., 2020), adversarial examples (Goodfellow et al.”
NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation (2022, methodology): “Under scenarios where test data are temporally correlated, however, naïvely adapting to the incoming batch of test samples [29, 33, 41, 44] could be problematic for both approaches: the batch is now more likely to (a) remove instance-wise variations that are actually useful to predict y, i.”
Tent: Fully Test-Time Adaptation by Entropy Minimization (2021, methodology): “For all test-time adaptation methods, we estimate these statistics during testing on the target data, as done in concurrent work on adaptation by normalization (Schneider et al., 2020; Nado et al., 2020).”
Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (2021, methodology): “We further compare T3A with other test-time adaptation methods when backbone networks employ the BN layer so that we can make a fair comparison with the prior BN-based methods [43, 52].”
Improved OOD Generalization via Adversarial Training and Pre-training (2021, background): “OOD generalization measures a model’s ability to extrapolate beyond the training distribution (Hendrycks & Dietterich, 2018), and has been widely explored in both CV (Recht et al., 2019; Schneider et al., 2020; Salman et al., 2020b) and NLP domains (Tu et al., 2020; Lohn, 2020).”