Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift

By Zachary Nado, Shreyas Padhy, David Sculley, A. D'Amour, Balaji Lakshminarayanan, Jasper Snoek. arXiv.org, 2020

332 citations recorded by the source corpus.

Covariate shift has been shown to sharply degrade both predictive accuracy and the calibration of uncertainty estimates for deep learning models. This is worrying, because covariate shift is prevalent in a wide range of real world deployment settings. However, in this paper, we note that frequently there exists the potential to access small unlabeled batches of the shifted data just before prediction time. This interesting observation enables a simple but surprisingly effective method which we call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. Using this one line code change, we achieve state-of-the-art on recent covariate shift benchmarks and an mCE of 60.28\% on the challenging ImageNet-C dataset; to our knowledge, this is the best result for any model that does not incorporate additional data augmentation or modification of the training pipeline. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness (e.g. deep ensembles) and combining the two further improves performance. Our findings are supported by detailed measurements of the effect of this strategy on model behavior across rigorous ablations on various dataset modalities. However, the method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift, and is therefore worthy of additional study. We include links to the data in our figures to improve reproducibility, including a Python notebooks that can be run to easily modify our analysis at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/69bad7481b4c7dab2f0ac7c518c2be7256e699bf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “This idea was further efﬁciently approximated by stochastic batch normalization [210] and exploited in prediction-time batch normalization [211].”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “batch normalization calibration PredBN [13], [370], InstCal [371] model optimization TTT [8], GeOS [365], MEMO [14] meta-learning MLSR [372], Full-OSHOT [373] input adaptation TPT [374], TTA-DAE [375] dynamic inference EMEA [376], DRM [377]”
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift (2020, methodology): “We evaluate the general approach of using test batches to compute batch normalization (BN) statistics, which has been proposed in several prior works [36, 29, 47, 58].”
Efficient Test-Time Model Adaptation without Forgetting (2022, methodology): “…address this, fully test-time adaptation methods have been proposed to adapt a model with only test data, including batchnorm statistics adaptation (Nado et al., 2020; Schneider et al., 2020; Khurana et al., 2021), test-time entropy minimization (Wang et al., 2021; Fleuret et al., 2021),…”
Tent: Fully Test-Time Adaptation by Entropy Minimization (2021, methodology): “Schneider et al. (2020); Nado et al. (2020) estimate target statistics during testing to improve generalization.”
NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation (2022, methodology): “Under scenarios where test data are temporally correlated, however, naïvely adapting to the incoming batch of test samples [29, 33, 41, 44] could be problematic for both approaches: the batch is now more likely to (a) remove instance-wise variations that are actually useful to predict y, i.”
MEMO: Test Time Robustness via Adaptation and Augmentation (2021, methodology): “Some methods update the model using batches or even entire datasets of test inputs, such as by computing batch normalization (BN) statistics on the test set (Li et al., 2017; Kaku et al., 2020; Nado et al., 2020; Schneider et al., 2020), or minimizing the (conditional) entropy of model predictions a”
Towards Stable Test-Time Adaptation in Dynamic Wild World (2023, methodology): “In TTA, prior methods often conduct adaptation on pre-trained models with batch normalization (BN) layers (Ioffe & Szegedy, 2015), and most of them are built upon BN statistics adaptation (Schneider et al., 2020; Nado et al., 2020; Khurana et al., 2021; Wang et al., 2021; Niu et al., 2022a; Hu et al”
Improving robustness against common corruptions by covariate shift adaptation (2020, background): “The ImageNet-C benchmark [2] consists of 15 test corruptions and four hold-out corruptions which are applied with five different severity levels to the 50 000 test images of the ILSVRC2012 subset of ImageNet [8].”