InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions

By Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li and 2 more. Computer Vision and Pattern Recognition, 2022

1,149 citations recorded by the source corpus.

Compared to the great progress of large-scale vision transformers (ViTs) in recent years, large-scale models based on convolutional neural networks (CNNs) are still in an early state. This work presents a new large-scale CNN-based foundation model, termed InternImage, which can obtain the gain from increasing parameters and training data like ViTs. Different from the recent CNNs that focus on large dense kernels, InternImage takes deformable convolution as the core operator, so that our model not only has the large effective receptive field required for downstream tasks such as detection and segmentation, but also has the adaptive spatial aggregation conditioned by input and task information. As a result, the proposed InternImage reduces the strict inductive bias of traditional CNNs and makes it possible to learn stronger and more robust patterns with large-scale parameters from massive data like ViTs. The effectiveness of our model is proven on challenging benchmarks including ImageNet, COCO, andADE20K. It is worth mentioning that InternImage-H achieved a new record 65.4 mAP on COCO test-dev and 62.9 mIoU on ADE20K, outperforming current leading CNNs and ViTs.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/26c80bd65baa90f5b18157de4951f4eb0b62ab69

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks (2023, positive): “We implement two variants of VisionLLM with two image backbones, i.e. , ResNet [21] and InternImage-H [51].”
EVA-02: A Visual Representation for Neon Genesis (2023, background): “6 InternImage-XL [124] 335M 6402 IN-21K 55.”
BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision (2022, methodology): “We conduct experiments with multiple types of back-bones: ResNet [8], DLA [39], VoVNet [13], and InternIm-age [34].”
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (2024, neutral): “Another line of works focuses on improving traditional 2D ConvNets with more advanced settings [41, 67].”
UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition (2023, background): “…UniRepLKNet outperforms Swin, ConvNeXt, RepLKNet, and SLaK, which are representatives of ViTs, modern medium-kernel ConvNets, and existing large-kernel ConvNets, respectively, and shows comparable performance to InternImage [82], which is a latest powerful architecture with deformable convolution.”
DETRs with Collaborative Hybrid Assignments Training (2022, methodology): “6% AP on COCO test-dev, outperforming the previous best model InternImage-G [33] with 3 billion parameters.”
MambaOut: Do We Really Need Mamba for Vision?* (2024, neutral): “…under the spirit of RNN. Neverthe-less, their experiments show that the SSM based models for vision, in reality, lead to underwhelming performance compared with state-of-the-art convolutional [23, 31, 39, 54, 58, 70, 86, 98, 100, 101] and attention-based models [19, 24, 51, 76, 80, 81, 100, 103].”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, positive): “…attempts have been made to address these challenges, including enhancing model architectures [220, 232, 5, 172, 157, 210], scaling vision encoders [252, 66, 36, 293, 185] and language models [231, 235, 64, 19, 229, 221, 62], incorporating more diverse and high-quality datasets [124, 234, 25,…”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “As depicted in Figure 3, unlike traditional vision-only back-bones [43, 71, 114] and dual-encoder models [51, 89, 99], the proposed InternVL is designed with a vision encoder InternViT-6B and a language middleware QLLaMA.”
Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment (2022, result): “Models such as EVA [9] and InterImage-H [28], with larger foundation models (ViT-giant [32] or InterImage-H [28]) and more data [8, 4, 23, 37, 26, 21], give higher results ( 64 .”
VideoChat: chat-centric video understanding (2023, neutral): “Ultimately, we rely on a pretrained LLM to address user-specified tasks by responding to questions based on video text descriptions.”
On the Challenges and Perspectives of Foundation Models for Medical Image Analysis (2023, methodology): “[2023], Wang et al. [2023b]. By combining the power of language and vision, these large-scale vision-language models have unlocked exciting possibilities for the future of medical foundation models.”
Griffin: Towards a Graph-Centric Relational Database Foundation Model (2025, unknown): “Foundation models have revolutionized domains such as natural language (Brown, 2020; Touvron et al., 2023), vision (Wang et al., 2023; Yuan et al., 2021), tabular data (Zhang et al., 2023b; Yang et al., 2024), and graphs (Liu et al., 2023; Tang et al., 2024), offering scalable and generalized…”