EVA-02: A Visual Representation for Neon Genesis

By Yuxin Fang, Yuxin Fang, Quan-Sen Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, Xinlong Wang, Yue Cao. Image and Vision Computing, 2023

551 citations recorded by the source corpus.

We launch EVA-02, a next-generation Transformer-based visual representation pre-trained to reconstruct strong and robust language-aligned vision features via masked image modeling. With an updated plain Transformer architecture as well as extensive pre-training from an open&accessible giant CLIP vision encoder, EVA-02 demonstrates superior performance compared to prior state-of-the-art approaches across various representative vision tasks, while utilizing significantly fewer parameters and compute budgets. Notably, using exclusively publicly accessible training data, EVA-02 with only 304M parameters achieves a phenomenal 90.0 fine-tuning top-1 accuracy on ImageNet-1K val set. Additionally, our EVA-02-CLIP can reach up to 80.4 zero-shot top-1 on ImageNet-1K, outperforming the previous largest&best open-sourced CLIP with only ~1/6 parameters and ~1/6 image-text training data. We offer four EVA-02 variants in various model sizes, ranging from 6M to 304M parameters, all with impressive performance. To facilitate open access and open research, we release the complete suite of EVA-02 to the community at https://github.com/baaivision/EVA/tree/master/EVA-02.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3049c992adbd56e29c4d957ee0c4e9d05fe3c6d1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, methodology): “Specifically, we use 2D RoPE in at each layer of our model, following the implementation of [Lu et al. 2023; Fang et al. 2023].”
Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection (2023, background): “As shown in Tab. by 1.0% mAP and 1.7% NDS. Scaling up the backbone to ViT-Large [7, 9], StreamPETR achieves 62.0% of mAP, 67.6% of NDS, and 25.8% of mAOE.”
DETRs with Collaborative Hybrid Assignments Training (2022, methodology): “This large-scale backbone ViT-L [7] is pre-trained using a selfsupervised learning method (EVA-02 [8]).”
GLaMM: Pixel Grounding Large Multimodal Model (2023, methodology): “For Level-1, we use Tag2Text [14] and RAM [58] for image tagging, Co-DETR [62], EVAv02 [7], OWL-ViT [33], and POMP [40] for object localization, GRiT [48] and GPT4RoI [57] for attribute generation, and MiDAS [39] for depth estimation.”
YOLOv12: Attention-Centric Real-Time Object Detectors (2025, unknown): “Although attention-centric vision transformer (ViT) architectures have been proven to possess stronger modeling capabilities, even in small models [20, 21, 25, 50], most architectural designs continue to focus primarily on CNNs.”
GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment (2023, methodology): “We compare the ViT-B/32 model against the larger ViT-L/14 model [31], an even larger ViT-H/14 model from OpenCLIP [8], and the EVA-02-CLIP model from [12].”
EVA-CLIP: Improved Training Techniques for CLIP at Scale (2023, methodology): “…that can significantly reduce training costs, stabilize the training process and improve zero-shot performance, including initialize CLIP with pre-trained EVA [20, 19] representations, the LAMB [52] optimizer, randomly dropping input tokens [33], and a speedup trick named flash attention [15].”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, background): “Although a recent study [33] shows that including unlabeled images from the development / test set for MIM pre-training has minimal impact on the final performance, we conduct a more rigorous analysis of the LVIS and COCO performance gap to eliminate potential data contamination issues: We evaluate ”
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, unknown): “We employ an off-the-shelf LVIS detector (Fang et al., 2023) to extract up to 100 detection boxes per image.”
Nomic Embed Vision: Expanding the Latent Space (2024, methodology): “5 respectively (Nussbaum et al., 2024), and the vision encoder as EVA02-ViT B/16 (Fang et al., 2023b).”