CoAtNet: Marrying Convolution and Attention for All Data Sizes

By Zihang Dai, Hanxiao Liu, Quoc V. Le, Mingxing Tan. 2021

1,668 citations recorded by the source corpus.

Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks. In this work, we show that while Transformers tend to have larger model capacity, their generalization can be worse than convolutional networks due to the lack of the right inductive bias. To effectively combine the strengths from both architectures, we present CoAtNets(pronounced"coat"nets), a family of hybrid models built from two key insights: (1) depthwise Convolution and self-Attention can be naturally unified via simple relative attention; (2) vertically stacking convolution layers and attention layers in a principled way is surprisingly effective in improving generalization, capacity and efficiency. Experiments show that our CoAtNets achieve state-of-the-art performance under different resource constraints across various datasets: Without extra data, CoAtNet achieves 86.0% ImageNet top-1 accuracy; When pre-trained with 13M images from ImageNet-21K, our CoAtNet achieves 88.56% top-1 accuracy, matching ViT-huge pre-trained with 300M images from JFT-300M while using 23x less data; Notably, when we further scale up CoAtNet with JFT-3B, it achieves 90.88% top-1 accuracy on ImageNet, establishing a new state-of-the-art result.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Mixture-of-Experts with Expert Choice Routing (2022, background): “Scaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27].”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “ViT-G (Zhai et al., 2021) 90.45 – CoAtNet-7 (Dai et al., 2021) 4 and Section 3.3.2, while analogous results for BASIC (Pham et al., 2021) are in Appendix C.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “…increase, but their accuracy on the robustness benchmarks increase much less, or decrease. that a model which achieves about 91% zero-shot accuracy on ImageNet, i.e. , just slightly better than the state-of-the-art ImageNet-trained model (Dai et al., 2021), will also achieve the ideal robustness.”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, methodology): “, 2021) and CoAtNet (Dai et al., 2021), our model receives the raw image x ∈ RH×W×C and maps it into flattened 1D sequence of patches xp ∈ RTi×D as input for the transformer, whereD is the fixed hidden size of the transformer layers and Ti = HW P 2 is the length of the image tokens for a given patch”
Scaling Vision Transformers (2021, background): “The concurrent work [12] focuses on CNN and ViT hybrid architectures.”
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “Ever since [18] first showed good results when pre-training BiT on ImageNet-21k, more architecture works have mentioned using it for select few experiments [10, 33, 32, 7], with [26] arguing more directly for the use of ImageNet-21k.”
Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “Considering the superior modeling capability of Vision Transformers [21] recently demonstrated on computer vision tasks [57,84,26,92,17], we take the utilization of the potential of attention mechanism [74] to develop online CL.”
Architecture Matters in Continual Learning (2022, background): “Pooling layers not only add local translation invariances, which help in applications like object classiﬁcation (Krizhevsky et al., 2017; Dai et al., 2021), but also reduce the spatial resolution of the network features, resulting in the reduction of computational cost.”
Transformers in Vision: A Survey (2021, background): “Depthwise Convolution and self-Attention Networks (CoAtNets) [49] introduce a relative attention module (which combines depthwise convolutions and self-attention), and vertically stack convolution and attention layers.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “In addition, we also use 1D relative position bias for text [30] and 2D relative position bias for image [22, 62].”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, methodology): “To address the data hungry problem, previous large vision models typically utilize huge labelled data such as JFT-3B [17, 56, 80].”
Measuring Forgetting of Memorized Training Examples (2022, background): “Such settings are used when training large image models [16, 53], multimodal models [63] and language models [48, 14, 32, 84], the latter of which have come under signiﬁcant scrutiny due to privacy concerns [12, 4].”
MetaFormer Baselines for Vision (2022, methodology): “For instance, ConvFormer-M36 outperforms Swin-B [45]/CoAtNet-2 [12] by 1.”
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, background): “As a result, the device will need to save a dedicated set of model parameters for each task, which causes a huge amount of storage space, especially for today’s large models ( e.g. , ViT-G/14 [11] 1.8G, CoAtNet [5] 2.4G).”
Combined Scaling for Open-Vocabulary Image Classiﬁcation (2022, methodology): “Thus, for our image encoder, we use the largest CoatNet architecture (Dai et al., 2021) due to its proven large learning capacity.”