EVA: Exploring the Limits of Masked Visual Representation Learning at Scale

By Yuxin Fang, Wen Wang, Binhui Xie, Quan-Sen Sun, Ledell Yu Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, Yue Cao. 2022

1,015 citations recorded by the source corpus.

We launch EVA, a vision-centric foundation model to Explore the limits of Visual representation at scAle using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVIS dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can greatly stabilize the training and outperform the training from scratch counterpart with much fewer samples and less compute, providing a new direction for scaling up and accelerating the costly training of multi-modal foundation models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, background): “CLIP [43], OpenCLIP [22], EVA-CLIP [15], etc .) are already pretrained on web-scale image-text paired data.”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, positive): “For the frozen image encoder, we explore two state-of-the-art pre-trained vision transformer models: (1) ViT-L/14 from CLIP (Rad-ford et al., 2021) and (2) ViT-g/14 from EVA-CLIP (Fang et al., 2022).”
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, background): “, ViT (Radford et al., 2021; Fang et al., 2023)) to get the vision representation of the image.”
Reproducible Scaling Laws for Contrastive Language-Image Learning (2022, methodology): “This already resulted in numerous works using the datasets for training state-of-the-art language-vision models [63, 49, 27, 76, 19], validating the usage of those datasets for studying scaling laws in this work.”
YOLOv12: Attention-Centric Real-Time Object Detectors (2025, neutral): “Many attention-centric vision transformers are designed with the plain-style architectures [1, 18, 19, 21, 25, 51], while we retain the hierarchical design of the previous YOLO systems [3, 24, 28, 29, 32, 45–47, 53, 57, 58] and will demonstrate the necessity of this.”
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, methodology): “Vision-language models (VLMs), exemplified by CLIP [66] and its subsequent works [26, 36, 43, 44, 50, 75, 94], align image and text features in a unified embedding space.”
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition (2023, methodology): “The visual encoder in InternLMXComposer employs EVA-CLIP [17], an refined variant of the standard CLIP [68], enhanced with mask image modeling capabilities, to proficiently capture the visual nuances of the input image.”
Are aligned neural networks adversarially aligned? (2023, methodology): “Mini GPT-4 [Zhu et al., 2023] uses a pretrained Q-Former module from [Li et al., 2023] to project images encoded by EVA CLIP ViT-G/14 [Fang et al., 2022] to Vicuna’s [Chiang et al., 2023] text embedding space.”
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, positive): “Previous inﬂuential work like DINOv2 [29], ConvNeXt [25], and EVA [10] demonstrates how incorporating advanced design strategies can revitalize classic methods [16, 48].”
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models (2023, neutral): “…, we ﬁnd that Otter-Image, BLIVA, and BLIP2 cannot follow the user’s instructions well and generate wrong reasoning steps sometimes, They attain a much worse ranking in the arena than quantitative evaluation because responses with hallucination would have a negative inﬂuence on user experience.”
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, methodology): “In this work, we instantiate the model with the pre-trained EVA-G [17] for visual encoder and QFormer [14] for text decoder by default.”
EVA-02: A Visual Representation for Neon Genesis (2023, methodology): “EVA (1011M) EVA-02 (304M) Figure 1: Qualitative comparisons between EVA-02 (#params: 304M) and EVA (#params: 1011M) [44] pre-trained representations.”
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (2023, methodology): “In our experiments, we adopt four variations of BLIP-2 with the same image encoder (ViT-g/14 [31]) but different frozen LLMs, including FlanT5-XL (3B), FlanT5-XXL (11B), Vicuna-7B and Vicuna-13B.”
OtterHD: A High-Resolution Multi-modality Model (2023, background): “The increasing success of large models that focus on a single modality, such as language models [41, 40, 54, 52, 11] and vision models [43, 20], has sparked a recent surge in research exploring combinations of these models.”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “…JFT [134], or are aligned with the BERT series [52, 54, 70] using image-text pairs, lacking direct alignment with LLMs. Additionally, the prevalent vision models employed to connect with LLMs are still limited to around 1 billion parameters [37, 51], which also constrains the performance of VLLMs.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, positive): “…have been made to address these challenges, including enhancing model architectures [220, 232, 5, 172, 157, 210], scaling vision encoders [252, 66, 36, 293, 185] and language models [231, 235, 64, 19, 229, 221, 62], incorporating more diverse and high-quality datasets [124, 234, 25, 155], and…”
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (2024, neutral): “…efficiency on Ima-geNet [10] and various downstream tasks [39, 74] by introducing 2D priors, with the surge of large-scale visual pretraining [1, 5, 17] and multi-modality applications [3, 29, 35, 36, 40, 49], vanilla Transformer-style model strikes back to the center stage of computer vision.”
A Cookbook of Self-Supervised Learning (2023, methodology): “Recent work has pushed these vision-language systems to larger scales [Ding et al., 2021, Yuan et al., 2021, Singh et al., 2022, Wang et al., 2022d, Fang et al., 2022b], based on freely available image-caption pairs collected from the internet, such as in [Schuhmann et al., 2022].”