ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders

By Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In-So Kweon, Saining Xie. Computer Vision and Pattern Recognition, 2023

1,840 citations recorded by the source corpus.

Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2020s. For example, modern ConvNets, represented by ConvNeXt [33], have demonstrated strong performance in various scenarios. While these models were originally designed for supervised learning with ImageNet labels, they can also potentially benefit from self-supervised learning techniques such as masked autoencoders (MAE) [14]. However, we found that simply combining these two approaches leads to subpar performance. In this paper, we propose a fully convolutional masked autoencoder framework and a new Global Response Normalization (GRN) layer that can be added to the ConvNeXt architecture to enhance inter-channel feature competition. This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation. We also provide pre-trained ConvNeXt V2 models of various sizes, ranging from an efficient 3.7M-parameter Atto model with 76.7% top-1 accuracy on ImageNet, to a 650M Huge model that achieves a state-of-the-art 88.9% accuracy using only public training data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2218f1713d7f721ab76801063416ec9b11c7646f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition (2023, methodology): “…ADE20K semantic segmentation [98], and COCO object detection [44], outperforming the existing large-kernel ConvNets such as RepLKNet [19], SLaK [48], and recent powerful architectures including ConvNeXt V2 [85], FastViT [77], Swin V2 [51] and DeiT III [75] in terms of both accuracy and efﬁciency.”
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles (2023, methodology): “Above we compare to recent MAE-based works (Woo et al., 2023; Feichtenhofer et al., 2022).”
EVA-02: A Visual Representation for Neon Genesis (2023, neutral): “8 ConvNeXt V2-T [129] 29M 3842 13G ✓ 85.”
A Cookbook of Self-Supervised Learning (2023, methodology): “More recently, masked autoencoding pre-training strategies have been adapted for convolutional architectures to great effect, where they achieve competitive performance on downstream object detection and instance segmentation [Woo et al., 2023].”
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, methodology): “Maintaining the simplicity of pipeline without phoneme alignment, duration predictor, text encoder, and semantically infused codec model, F5-TTS leverages the Diffusion Transformer with ConvNeXt V2 [42] to better tackle text-speech alignment during in-context learning.”
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (2023, methodology): “Thanks to the strong universal representation learned by Florence-2 pre-training, we do not require longer training epochs, such as 36 epochs in [51, 81, 85, 86], or 100 epochs in [46], to achieve better results.”
YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information (2024, methodology): “…mainly uses repeated input data and maintains the information of the input data in an explicit way; (2) The use of masked modeling [1, 6, 9, 27, 71, 73]: it mainly uses reconstruction loss and adopts an implicit way to maximize the extracted features and retain the input information; and (3)…”