A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence

By Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, Ming-Hsuan Yang. Neural Information Processing Systems, 2023

356 citations recorded by the source corpus.

Text-to-image diffusion models have made significant advances in generating and editing high-quality images. As a result, numerous approaches have explored the ability of diffusion model features to understand and process single images for downstream tasks, e.g., classification, semantic segmentation, and stylization. However, significantly less is known about what these features reveal across multiple, different images and objects. In this work, we exploit Stable Diffusion (SD) features for semantic and dense correspondence and discover that with simple post-processing, SD features can perform quantitatively similar to SOTA representations. Interestingly, the qualitative analysis reveals that SD features have very different properties compared to existing representation learning features, such as the recently released DINOv2: while DINOv2 provides sparse but accurate matches, SD features provide high-quality spatial information but sometimes inaccurate semantic matches. We demonstrate that a simple fusion of these two features works surprisingly well, and a zero-shot evaluation using nearest neighbors on these fused features provides a significant performance gain over state-of-the-art methods on benchmark datasets, e.g., SPair-71k, PF-Pascal, and TSS. We also show that these correspondences can enable interesting applications such as instance swapping in two images.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/3ad0899e9fa71cc6c294db4e8061ff6bc9e22225

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (2023, unknown): “Even without providing object location information, it is found that the T2I model can align the semantic prompts with the corresponding regions in the image due to its underlying semantic segmentation capability [68].”
Style Injection in Diffusion: A Training-Free Approach for Adapting Large-Scale Diffusion Models for Style Transfer (2023, unknown): “2 (a), in SA-based style transfer, the content image patches ( query ) that share semantic similarities engage with a similar style ( key ), thereby maintaining the relationship among these content image patches after the transfer.”
TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, background): “Concurrent works demonstrated semantic appearance swapping using diffusion feature correspondences ([27, 55]).”