Deep ViT Features as Dense Visual Descriptors

By Shirzad Amir, Yossi Gandelsman, Shai Bagon, Tali Dekel. arXiv.org, 2021

415 citations recorded by the source corpus.

We study the use of deep features extracted from a pretrained Vision Transformer (ViT) as dense visual descriptors. We observe and empirically demonstrate that such features, when extractedfrom a self-supervised ViT model (DINO-ViT), exhibit several striking properties, including: (i) the features encode powerful, well-localized semantic information, at high spatial granularity, such as object parts; (ii) the encoded semantic information is shared across related, yet different object categories, and (iii) positional bias changes gradually throughout the layers. These properties allow us to design simple methods for a variety of applications, including co-segmentation, part co-segmentation and semantic correspondences. To distill the power of ViT features from convoluted design choices, we restrict ourselves to lightweight zero-shot methodologies (e.g., binning and clustering) applied directly to the features. Since our methods require no additional training nor data, they are readily applicable across a variety of domains. We show by extensive qualitative and quantitative evaluation that our simple methodologies achieve competitive results with recent state-of-the-art supervised methods, and outperform previous unsupervised methods by a large margin. Code is available in dino-vit-features.github.io.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3966121cb48c2195b5e6d63e3de4122284c67309

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data (2023, background): “[5] show that DINO encodes valuable semantic information about object parts.”
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (2024, background): “Self-supervised vision models (e.g., DINO [5, 118]), on the other hand, provide fine-grained pixel-level features useful for various vision and robotic tasks [31, 119–124], but there lack effective ways for interpreting open-world semantics pivotal for cross-task generalization.”
DINOv2: Learning Robust Visual Features without Supervision (2023, background): “Additionally, the features output by self-supervised models have been shown to exhibit various useful properties, and have enabled enabled a wide variety of applications (Amir et al., 2022; Tumanyan et al., 2022; Ofri-Amar et al., 2023; Hamilton et al., 2022).”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “Amir et al. [2021] also of DINO features.”
Emergent Correspondence from Image Diffusion (2023, background): “Meanwhile, recent works on self-supervised representation learning [10] has yielded strong per-pixel features that could be used to identify visual correspondence [82, 3, 10, 30].”
LERF: Language Embedded Radiance Fields (2023, background): “DINO has been shown to exhibit emergent object decomposition properties despite training on no labels[1], and additionally dis-tills well into 3D ﬁelds[20], making it a good candidate for grouping language in 3D without relying on labeled data or imparting too strict a prior.”
StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis (2023, background): “The network is lightweight, fast to evaluate, and encodes semantic information at high spatial resolution (Amir et al., 2021).”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “In a similar vein, features from image-only foundation models like DINO [7] have been employed as drop-in solutions for reasoning about visual concepts, such as classification, detection, segmentation, keypoint estimation [21, 22].”