On Robustness and Transferability of Convolutional Neural Networks

By Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, A. D'Amour, Dan I. Moldovan and 4 more. 2020

174 citations recorded by the source corpus.

Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts. However, several recent breakthroughs in transfer learning suggest that these networks can cope with severe distribution shifts and successfully adapt to new tasks from a few training examples. In this work we study the interplay between out-of-distribution and transfer performance of modern image classification CNNs for the first time and investigate the impact of the pre-training data size, the model scale, and the data preprocessing pipeline. We find that increasing both the training set and model sizes significantly improve the distributional shift robustness. Furthermore, we show that, perhaps surprisingly, simple changes in the preprocessing such as modifying the image resolution can significantly mitigate robustness issues in some cases. Finally, we outline the shortcomings of existing robustness evaluation datasets and introduce a synthetic dataset SI-SCORE we use for a systematic analysis across factors of variation common in visual data such as object size and position.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Safety, Robustness & Interpretability

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, background): “Moreover, Sun et al. (2017) study how CNN performance scales with dataset size, and Kolesnikov et al. (2020); Djolonga et al. (2020) perform an empirical exploration of CNN transfer learning from large scale datasets such as ImageNet-21k and JFT-300M.”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “Shifted performance evaluations form the backbone of empirical evaluations in the literature on robust machine learning and task adaptation (e.g., Hendrycks and Dietterich, 2019; Wang et al., 2019; Djolonga et al., 2020; Taori et al., 2020).”
Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers (2021, methodology): “As for CNNs, Djolonga et al. (2020) evaluate the impact of the model size and dataset size on robustness, where classes at train and test time are the same, but there is a distribution shift in the data, for example changes in the lighting of image samples.”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, background): “It was shown that scaling up model and dataset sizes simultaneously leads to dramatic improvements in transfer effectiveness [20, 32, 68] and robustness [17].”
Toward Causal Representation Learning (2021, background): “So far, there has been no deﬁnitive consensus on how to solve these problems, although progress has been made using data augmentation, pretraining, self-supervision, and architectures with suitable inductive biases with respect to a perturbation of interest [60], [64], [137], [170], [206], [233].”
Towards Causal Representation Learning (2021, background): “This does not mean obtaining more examples from P (X,Y ), but training on a richer dataset [244, 53], for example, through pre-training on a huge and diverse corpus [196, 54, 112, 137, 59, 35, 45, 253].”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “For example, models can be robust to image corruptions but not to shifts across datasets (Taori et al., 2020; Djolonga et al., 2020), and a method that improves robustness on a standard vision dataset can actually consistently harm robustness on real-world satellite imagery datasets (Xie et al.,…”
Adversarial Robustness with Semi-Infinite Constrained Learning (2021, background): “[14] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D’Amour, Dan Moldovan, et al.”
Scaling Vision Transformers to 22 Billion Parameters (2023, result): “We can conﬁrm results from (Taori et al., 2020; Djolonga et al., 2021; Kolesnikov et al., 2020) that scaling the model increases out-of-distribution performance in line with the improvements on ImageNet.”
The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning (2021, methodology): “We also study a series of BiT models that were pre-trained in [6] on different amounts of data from JFT.”
How Well Do Sparse ImageNet Models Transfer? (2021, background): “This has been further extended by Djolonga et al. [10], who concluded that increasing the scale of the original model and dataset sig-niﬁcantly improves out-of-distribution and transfer performance, despite having marginal impact on the original accuracy.”
Toward Transformer-Based Object Detection (2020, background): “ObjectNet has proven to be useful for the evaluation of the generalization and robustness of image classification models [19, 11, 35].”