EVA-CLIP: Improved Training Techniques for CLIP at Scale

By Quan-Sen Sun, Yuxin Fang, Ledell Yu Wu, Xinlong Wang, Yue Cao. arXiv.org, 2023

929 citations recorded by the source corpus.

Contrastive language-image pre-training, CLIP for short, has gained increasing attention for its potential in various scenarios. In this paper, we propose EVA-CLIP, a series of models that significantly improve the efficiency and effectiveness of CLIP training. Our approach incorporates new techniques for representation learning, optimization, and augmentation, enabling EVA-CLIP to achieve superior performance compared to previous CLIP models with the same number of parameters but significantly smaller training costs. Notably, our largest 5.0B-parameter EVA-02-CLIP-E/14+ with only 9 billion seen samples achieves 82.0 zero-shot top-1 accuracy on ImageNet-1K val. A smaller EVA-02-CLIP-L/14+ with only 430 million parameters and 6 billion seen samples achieves 80.4 zero-shot top-1 accuracy on ImageNet-1K val. To facilitate open access and open research, we release the complete suite of EVA-CLIP to the community at https://github.com/baaivision/EVA/tree/master/EVA-CLIP.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a08b7123a7158f1a7fbbc18e8b5aaebd47980ecf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

Citations in the corpus

Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “Speciﬁcally, Sun et al. [70] present a method to improve training stability and reduce computational costs, including improved initialization, better optimizer, and random masking of images [66].”
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, methodology): “Given a video input V ∈ R T × H × W × 3 , the encoder first uses a pre-trained image encoder, i.e. ViT [35, 42], to encode each frame and obtain frame features.”
Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, neutral): “A key benefit of assembling our architecture from plain ViT encoders over custom encoders is the abundance of pretrained ViT-based backbones that can be harnessed (Oquab et al., 2024; Peng et al., 2022b; Sun et al., 2023).”
BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, methodology): “…large language models (LLMs) [13, 21,62,75,90], a sequence of studies explore multimodal LMMs that can jointly understand vision and language information and generate textual answers through adding a modality adaption structure between a frozen visual encoder [27,64,69] and a frozen LLM [75,90].”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “Following previous meth-ods [89, 99, 117], we report the top-1 accuracy and the mean of top-1 and top-5 accuracy on Kinetics-400/600/700 [15– 17].”
LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “The rapid development of Multi-modal Large Language Models (MLLMs) has led to significant advances in vision-language understanding [26, 37, 38], driven by break-throughs in both large language models [3, 8, 25] and visual encoders [30, 35, 46].”
CogAgent: A Visual Language Model for GUI Agents (2023, positive): “…low-resolution input branch, the high-resolution cross-module adopts a much smaller pre-trained vision encoder (visual encoder of EVA2-CLIP-L [35] in our implementation, 0.30B parameters), and uses cross-attention of a small hidden size to fuse high-resolution image features with every layer…”
Transfer Visual Prompt Generator across LLMs (2023, neutral): “[54] NF-ResNet-50 [6] NF-ResNet-50 GPT-2-like † [42] No Flamingo [2] NFNet-F6 [6]+Resampler [21] Resampler Chinchilla [18] Xattn-Dense PaLM-E [16] ViT [15] / OSRT [49] All PaLM [11] No BLIP-2 [30] EVA-CLIP [52] + Q-Former [30] Q-Former OPT [60] / Flan-T5 [12] No Training Paradigm.”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “…understanding component, a stronger vision encoder can be chosen without worrying about whether the encoder is capable of handling vision generation tasks, such as EVA-CLIP [71], InternViT [13], etc. (2) To handle high-resolution images, dynamic high-resolution techniques [48] can be used.”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “…in building video foundation models (ViFM) include video-text contrastive learning [Li and Wang, 2020, Xu et al., 2021, Wang et al., 2022], masked video modeling [Tong et al., 2022, Wang et al., 2023b, 2022, Fu et al., 2021], and next token prediction [Alayrac et al., 2022, Sun et al., 2023c,a].”
VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model (2025, neutral): “However, a key limitation of the VLMs at that time lies in their constrained image input resolution, which is restricted by the capabilities of their underlying vision encoders [43, 47, 57].”
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (2024, methodology): “At their core, most MLLMs [8, 31, 71] are built on pre-trained vision [43, 54] and language [59, 68, 69] models.”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, methodology): “Similar to SpatialVLM [15], we use a CLIP-based open-vocabulary classification model [61] to identify and exclude these unsuitable images.”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “EVA-CLIP [69] uses a distillation model with masked modeling to learn image-text joint representation.”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “…instead explore incorporating MLLMs for image synthesis using text-to-image models such as Stable Diffusion, and the objective is to generate conditional embeddings that align pretrained CLIP text ( i.e. , CLIP) or CLIP variant embeddings (Koh et al., 2023; Ge et al., 2023; Sun et al., 2023a;b).”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, methodology): “6, we first apply EVA-CLIP-g [68] akin to VideoChat, which achieves 6.9% higher accuracy with our instruction data (42.4% vs. 35.5% for original one in Tab.”
Generative Multimodal Models are In-Context Learners (2023, methodology): “Specifically, we leverage pre-trained EVA-02-CLIP-E-plus [70], LLaMA-33B [73] and SDXL [58] to initialize the Visual Encoder, Multimodal Modeling, and Visual Decoder, respectively.”
VideoChat: chat-centric video understanding (2023, positive): “It employs an architecture that combines both video and language foundation models with an addition learnable Video-Language Token Interface (VLTF).”