Are Transformers More Robust Than CNNs?

By Yutong Bai, Jieru Mei, Alan L. Yuille, Cihang Xie. 2021

345 citations recorded by the source corpus.

Transformer emerges as a powerful tool for visual recognition. In addition to demonstrating competitive performance on a broad range of visual benchmarks, recent works also argue that Transformers are much more robust than Convolutions Neural Networks (CNNs). Nonetheless, surprisingly, we find these conclusions are drawn from unfair experimental settings, where Transformers and CNNs are compared at different scales and are applied with distinct training frameworks. In this paper, we aim to provide the first fair&in-depth comparisons between Transformers and CNNs, focusing on robustness evaluations. With our unified training setup, we first challenge the previous belief that Transformers outshine CNNs when measuring adversarial robustness. More surprisingly, we find CNNs can easily be as robust as Transformers on defending against adversarial attacks, if they properly adopt Transformers' training recipes. While regarding generalization on out-of-distribution samples, we show pre-training on (external) large-scale datasets is not a fundamental request for enabling Transformers to achieve better performance than CNNs. Moreover, our ablations suggest such stronger generalization is largely benefited by the Transformer's self-attention-like architectures per se, rather than by other training setups. We hope this work can help the community better understand and benchmark the robustness of Transformers and CNNs. The code and models are publicly available at https://github.com/ytongbai/ViTs-vs-CNNs.

Subfields: Artificial Intelligence, Model Architectures, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Safety, Robustness & Interpretability

Citations in the corpus

Towards Robust Vision Transformer (2021, methodology): “As a training augmentation of ViTs, RandAugment adopts conflicted augmentation with ImageNet-C and may cause the unfairness of the comparison proposed by [1].”
Delving Deep into the Generalization of Vision Transformers under Distribution Shifts (2021, background): “Though recent works [1, 5, 20, 21] have studied several properties of ViTs, the generalization of ViTs is still under explored.”
Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap (2022, background): “This finding has been observed in other domains and remains under active investigation by the community [65].”
Generating Molecular Conformer Fields (2023, background): “On the other hand, recent literature shows that models that can learn these symmetries from data rather than explicitly en-3 forcing them ( e . g . transformers vs CNNs) tend to perform better as they are more amenable to optimization (Bai et al., 2021).”
Diffusion Models for Adversarial Purification (2022, background): “Note that for the adversarial training baselines, the training recipes for CNNs cannot be directly applied to ViTs due to the over-regularization issue (Bai et al., 2021).”