A Fourier Perspective on Model Robustness in Computer Vision

By Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin Dogus Cubuk, Justin Gilmer. 2019

638 citations recorded by the source corpus.

Achieving robustness to distributional shift is a longstanding and challenging goal of computer vision. Data augmentation is a commonly used approach for improving robustness, however robustness gains are typically not uniform across corruption types. Indeed increasing performance in the presence of random noise is often met with reduced performance on other corruptions such as contrast change. Understanding when and why these sorts of trade-offs occur is a crucial step towards mitigating them. Towards this end, we investigate recently observed trade-offs caused by Gaussian data augmentation and adversarial training. We find that both methods improve robustness to corruptions that are concentrated in the high frequency domain while reducing robustness to corruptions that are concentrated in the low frequency domain. This suggests that one way to mitigate these trade-offs via data augmentation is to use a more diverse set of augmentations. Towards this end we observe that AutoAugment, a recently proposed data augmentation policy optimized for clean accuracy, achieves state-of-the-art robustness on the CIFAR-10-C benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “Recent work has connected lack of robustness to computer vision models’ encoding counterintuitive features (Ilyas et al., 2019; Geirhos et al., 2019; Yin et al., 2019; Wang et al., 2020).”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “It is also interesting to note (from Figure 7) that PGD models actually see a drop in robustness to low frequency corruptions such as contrast, a phenomenon also observed in [107].”
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, background): “, 2020 [14] find that many recent language models are already robust to many forms of distribution shift, while others [37, 10] find that vision models are largely fragile and argue that data augmentation offers one solution.”
Natural Adversarial Examples (2019, background): “Although some data augmentation techniques are purported to greatly improve robustness to distribution shifts [30, 54], their lackluster results on IMAGENET-A show they do not improve robustness on some distribution shifts.”
Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “Training a machine learning model with a learned data augmentation policy may significantly improve image classification [4, 19, 15], object detection [51], model robustness [25, 46, 34], and semi-supervised learning image classification [44].”
When do neural networks outperform kernel methods? (2020, methodology): “Finally, the Fourier decomposition of images is a simple proxy for the decomposition of the covariate vector x into its low-dimensional dominant component (low frequency) and high-dimensional component (high frequency) [33].”
Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, background): “Other theories focus on the Fourier perspective of the robustness [67, 68], showing that adversarial training might be preventing the network from learning the high frequency signals of the input image.”
AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty (2019, methodology): “For further details on Fourier sensitivity analysis, we refer the reader to Section 2 of Yin et al. (2019).”
Adversarial Examples Improve Image Recognition (2019, background): “Moreover, such trained models are much more robust to high frequency noise [47].”
Learning Data Augmentation Strategies for Object Detection (2019, methodology): “Instead of cropping to focus on parts of the image, some methods randomly erase or add noise to patches of images for improved accuracy [9, 53, 13], robustness [50, 12], or both [29].”
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation (2019, methodology): “In an attempt to understand Patch Gaussian ’s performance, we perform a frequency-based analysis of models trained with various augmentations using the method introduced in [50].”
Out-of-Distribution Generalization via Risk Extrapolation (REx) (2020, background): “, 2018) and non-adversarial (Hendrycks & Dietterich, 2019; Yin et al., 2019) robustness, causality (Arjovsky et al.”
Understanding and Mitigating the Tradeoff Between Robustness and Accuracy (2020, result): “This is similar to the recent observation that adversarial training with `∞ perturbations encourages neural networks to fit the high frequency components of the signal while compromising on the low-frequency components (Yin et al., 2019).”
Causally motivated shortcut removal using auxiliary labels (2021, methodology): “Discouraging shortcut learning using data augmentation (e.g., rotation, translation, cropping) has been suggested by multiple authors [Hendrycks et al., 2020, Yin et al., 2019, Lyle et al., 2020, Lopes et al., 2019, Cubuk et al., 2018].”
Inception Transformer (2022, background): “Unlike ViTs, they cover more local information through local convolution within the receptive fields, thus effectively extracting high-frequency representations [19, 20].”
Continual Test-Time Domain Adaptation (2022, background): “A number of works have also shown that data augmentation [52] during training [14, 16, 32, 69] and during testing [1, 40, 73] can improve model robustness and generalizability.”
Watch Your Up-Convolution: CNN Based Generative Deep Neural Networks Are Failing to Reproduce Spectral Distributions (2020, background): “In [55], ﬁrst observations have been made that hint towards the importance of the power spectra on model robustness, again for image classiﬁcation.”
Increasing the robustness of DNNs against image corruptions by playing the Game of Noise (2020, background): “Yin et al. [2020] study the Fourier properties of common corruptions and link them to the robustness of diﬀerently trained classiﬁers.”