DINOv2: Learning Robust Visual Features without Supervision

By Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby and 20 more. Trans. Mach. Learn. Res., 2023

9,372 citations recorded by the source corpus.

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5a9cb1b3dc4655218b3deeaf4a2417a9a8cd0891

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, neutral): “Adversarial Diffusion Distillation (ADD) [49] provides the current state-of-the-art method for single-step synthesis: By leveraging a pretrained DINOv2 [36] feature extractor as the backbone of the discriminator, ADD manages to distill SDXL [38] into a single-step, real-time text-to-image model.”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “On tasks that require a fine-grained understanding of motion (Something-Something-v2), the V-JEPA models provide a major improvement (over +21 points) compared to large-scale image baselines, such as DINOv2, OpenCLIP, and I-JEPA.”
A foundation model for clinical-grade computational pathology and rare cancers detection (2024, positive): “33 with the following changes: a teacher temperature schedule of 0.04–0.07 in 186,000 iterations and a reciprocal square root learning rate schedule with a warmup of 495,000 iterations (instead of 100,000) and linear cooldown to 0.0 for the last 819,200 iterations 29 .”
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, neutral): “It consists of powerful vision transformers, DINOv2 [49] and SigLIP [74], pre-trained on Internet-scale image data, to capture rich visual features and a comprehensive semantic understanding of the observations.”
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, neutral): “For the mixed four vision encoders in SPHINX, we only preserve two of them, i.e., CLIP-ConvNeXt (Liu et al., 2022) and DINOv2 (Oquab et al., 2023).”
UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, neutral): “The architecture integrates a fused visual encoder derived from SigLip [90] and DINOv2 [62], a projection layer to align visual embeddings with the language modality, and the LLaMA-2 large language model (LLM) [75].”
Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, positive): “Following this analysis, we pick ViT-L DINOv2 for our encoder backbones.”
SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (2023, background): “Results show our F-ST encoder exhibits superior performance with substantially fewer parameters than 3D structure.”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, background): “Driven by the progress in large-scale model pre-training, similar foundation models for images were also explored in another line of work [5, 48], which aim to extract significant class-agnostic features from images.”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “Efficient-3DiM finetunes the SD model with a stronger vision transformer DINO v2 [35].”
A Cookbook of Self-Supervised Learning (2023, methodology): “, 2022a] and DINOV2 [Oquab et al., 2023] employ a mix of masked image modeling and more classical approaches such as self-distillation.”
Vision Transformers Need Registers (2023, methodology): “DINOv2 (Oquab et al., 2023) is a self-supervised method for learning visual features, following the DINO work.”
Foundation models in robotics: Applications, challenges, and the future (2023, positive): “Many foundation models (e.g., the GPT models, the Dall-E models) can only be accessed this way, while others are usually accessed this way, but can also be downloaded and run locally with sufﬁcient local computing power (such as SAM [59], LLaMA [214], and DINOv2 [107]).”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “Following common practices [30, 44, 84], we adopt the linear probing evaluation, i.e . training a linear classifier while keeping the backbone frozen.”
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, methodology): “Each extracted mask m t,i is then passed to a visual feature extractor (CLIP [31], DINO [54]) to obtain a visual descriptor f t,i = Embed ( I rgb t , m t,i ) .”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, methodology): “[41] introduce DINOv2 by scaling up their training data and the model size and observe significant performance boost on numerous single-image tasks, such as segmentation and depth.”
EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, methodology): “In the visual-enhanced perception part, the DINOv2 ViT-L/14 [94] is adopted as the ViT encoder, and the frozen CLIP ConvNeXt-L [95] is utilized as the CNN encoder.”
A whole-slide foundation model for digital pathology from real-world data (2024, positive): “Prov-GigaPath tile encoder used the ViT model architecture with standard DINOv2 settings 24 .”