Unleashing Text-to-Image Diffusion Models for Visual Perception

By Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, Jiwen Lu. IEEE International Conference on Computer Vision, 2023

350 citations recorded by the source corpus.

Diffusion models (DMs) have become the new trend of generative models and have demonstrated a powerful ability of conditional synthesis. Among those, text-to-image diffusion models pre-trained on large-scale image-text pairs are highly controllable by customizable prompts. Unlike the unconditional generative models that focus on low-level attributes and details, text-to-image diffusion models contain more high-level knowledge thanks to the vision-language pre-training. In this paper, we propose VPD (Visual Perception with pre-trained Diffusion models), a new framework that exploits the semantic information of a pre-trained text-to-image diffusion model in visual perception tasks. Instead of using the pre-trained denoising autoencoder in a diffusion-based pipeline, we simply use it as a backbone and aim to study how to take full advantage of the learned knowledge. Specifically, we prompt the denoising decoder with proper textual inputs and refine the text features with an adapter, leading to a better alignment to the pre-trained stage and making the visual contents interact with the text prompts. We also propose to utilize the cross-attention maps between the visual features and the text features to provide explicit guidance. Compared with other pre-training methods, we show that vision-language pre-trained diffusion models can be faster adapted to downstream visual perception tasks using the proposed VPD. Extensive experiments on semantic segmentation, referring image segmentation, and depth estimation demonstrate the effectiveness of our method. Notably, VPD attains 0.254 RMSE on NYUv2 depth estimation and 73.3% oIoU on RefCOCO-val referring image segmentation, establishing new records on these two benchmarks. Code is available at https://github.com/wl-zhao/VPD.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c09ca9da1fce13b1560f45c38321c7bb971f13fc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, background): “In con-strast to GAN [4], some new works show that diffusion models can be employed in some 3D tasks, such as optical flow estimation [10,54], view synthesis [31,53,58], depth estimation [21,24,81], and normal estimation [32,35,45].”
Emergent Correspondence from Image Diffusion (2023, background): “Some recent works [5, 90, 96] fine-tune either the U-Net or the attached head for dense prediction tasks and yield better performance.”
Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (2023, neutral): “Finally, VPD [64] employs a pretrained Stable Diffusion model as an image feature extractor with additional text input.”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, methodology): “As shown in Table 3 of NYUv2 [54], our model outperforms the previous best method VPD [86] remarkably, improving the δ 1 ( ↑ ) from 0.964 → 0.984 and AbsRel ( ↓ ) from 0.069 to 0.056.”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, methodology): “Several recent works [4, 65, 70] have used the pre-trained SD features for single-image tasks, such as semantic segmentation and depth perception.”