A ConvNet for the 2020s
By Zhuang Liu, Hanzi Mao, Chaozheng Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie. 2022
8,792 citations recorded by the source corpus.
The “Roaring 20s” of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model. A vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers (e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide variety of vision tasks. However, the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive biases of convolutions. In this work, we reexamine the design spaces and test the limits of what a pure ConvNet can achieve. We gradually “modernize” a standard ResNet toward the design of a vision Transformer, and discover several key components that contribute to the performance difference along the way. The outcome of this exploration is a family of pure ConvNet models dubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Dacheng Tao
- Quoc V. Le
- Trevor Darrell
- Yoshua Bengio
- Jiashi Feng
- Alan L. Yuille
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Yi Yang
- Alan L. Yuille
- Chunhua Shen
In Model Architectures
Citations in the corpus
- Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “For CNN model families (ResNets, EfficientNets, ResNexts, and ConvNeXts), the student accuracy is generally correlated with the teacher accuracy.”
- GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, methodology): “We resize spatial size into 256 × 256 and use the convnext-tiny [9] as the backbone to output a feature with spatial size as 8 × 8 , which then is flattened into 64 grounding tokens.”
- Visual Prompt Tuning (2022, background): “Backbone Pre-trained Objective Pre-trained Dataset # params (M) Feature dim d Batch Size Pre-trained Model ViT-B/16 [16] Supervised ImageNet-21k 85 768 2048 / 1280 / 128 / 128 / 64 checkpoint ViT-L/16 [16] 307 1024 2048 / 640 / 64 / 64 / 32 checkpoint ViT-H/14 [16] 630 1280 1024 / 240 / 28 / 28 / 14”
- OneFormer: One Transformer to Rule Universal Image Segmentation (2022, methodology): “8 — kMaX-DeepLab‡‡ [60] ConvNeXt-L† [39] 232M 1302G 256 1281×1281 100k 50.”
- First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, methodology): “To stress the importance of adaptation on datasets far from ImageNet, we compare FSA-FiLM with an EfficientNet-B0 backbone to the no-adaptation method with a ConvNext [14] pre-trained on Imagenet-22k.”
- Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, background): “We demonstrate that ConvNeXt blocks can more effectively model spatially local input patterns.”
- It's Raw! Audio Generation with State-Space Models (2022, neutral): “Compared to Gu et al. [13], we add additional pointwise linear layers after the S4 layer in the style of the feed-forward network in Transformers or the inverted bottleneck layer in CNNs [27].”
- Focal Modulation Networks (2022, background): “…are integrated to SA and bring significant gains [74, 22, 39, 20] or the vice versa [64]; ( ii ) ResNets have closed the gap to ViTs using similar data augmentation and regularization strategies [73], and replacing SA with (dynamic) depth-wise convolution [25, 47] can also slightly surpass Swin.”
- Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks (2023, methodology): “7, e.g ., Mobile-ViT, EdgeNext, PVT, CycleMLP, ConvNeXt, Swin, etc., also adopt such advanced training techniques (ADT).”
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (2024, neutral): “…composed of several key components, including language bind encoder f vM (Zhu et al., 2023a) for extracting features from raw visual inputs (e.g., images or videos), a language decoder model f L such as Vicuna (Touvron et al., 2023), a visual projection layer f P , and a word embedding layer f T .”
- Visual attention network (2022, methodology): “ConvNeXt [21] is a special CNN which absorbs the some advantages of ViTs such as large receptive field (7×7 convolution) and advanced training strategy (300 epochs, data augmentation, etc).”
- MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, methodology): “Moreover, MedViT-L outperforms the ConvNext-B by 0.8%, which has approximately two times more parameters than ours.”
- EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, background): “The detailed model configurations are (arch-model size-img resolution-data): ConvNeXt-XL-384px21K [66], SwinV2-L-384px-21K [65], MAE-H-448px-1K [39], DeiT3-L384px-21K [96], EfficientNet-L2&NS-800px-JFT300M [110], BEiTv2-L224px-21K [70], BEiT-L-512px-21K [5], EVA-g-336px-merged30M&21K.”
- Rewrite the Stars (2024, positive): “Since AlexNet [33], a myriad of deep networks [4, 23, 32, 37, 49] have emerged, each building on the other.”
- EfficientAD: Accurate Visual Anomaly Detection at Millisecond-Level Latencies (2023, background): “[35] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.”
- Towards Backward-Compatible Continual Learning of Image Compression (2024, methodology): “Our model employs the ConvNeXt module [35] as the basic building blocks.”
- MaxViT: Multi-Axis Vision Transformer (2022, methodology): “Throughput is measured on a single V100 GPU with batch size 16, following [54,55,77].”