Emergent Correspondence from Image Diffusion

By Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, Bharath Hariharan. Neural Information Processing Systems, 2023

511 citations recorded by the source corpus.

Finding correspondences between images is a fundamental problem in computer vision. In this paper, we show that correspondence emerges in image diffusion models without any explicit supervision. We propose a simple strategy to extract this implicit knowledge out of diffusion networks as image features, namely DIffusion FeaTures (DIFT), and use them to establish correspondences between real images. Without any additional fine-tuning or supervision on the task-specific data or annotations, DIFT is able to outperform both weakly-supervised methods and competitive off-the-shelf features in identifying semantic, geometric, and temporal correspondences. Particularly for semantic correspondence, DIFT from Stable Diffusion is able to outperform DINO and OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k benchmark. It even outperforms the state-of-the-art supervised methods on 9 out of 18 categories while remaining on par for the overall performance. Project page: https://diffusionfeatures.github.io

Subfields: Computer Vision and Pattern Recognition, Pediatrics, Perinatology and Child Health, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/f421b314aaff48e463507034691cfdd3f93cd4c2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Song Bai
2. Noah A. Smith
3. Wei-Chiu Ma
4. Jiachun Pan
5. Ranjay Krishna
6. Yujun Shi
7. Vincent Y. F. Tan
8. Xingyu Fu

In Radiology, Nuclear Medicine and Imaging

1. Noah A. Smith

Citations in the corpus

• BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, methodology): “Specifically, DepthAnything [83] and DIFT [70] achieve human-level performance on depth estimation and visual correspondence, whereas multimodal LLMs fail miserably.”
• DragDiffusion: Harnessing Diffusion Models for Interactive Point-Based Image Editing (2023, methodology): “To compute the MD, we first employ DIFT [45] to identify points in the edited images corresponding to the handle points in the original image.”