Label-Efficient Semantic Segmentation with Diffusion Models

By Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, Artem Babenko. 2021

723 citations recorded by the source corpus.

Denoising diffusion probabilistic models have recently received much research attention since they outperform alternative approaches, such as GANs, and currently provide state-of-the-art generative performance. The superior performance of diffusion models has made them an appealing tool in several applications, including inpainting, super-resolution, and semantic editing. In this paper, we demonstrate that diffusion models can also serve as an instrument for semantic segmentation, especially in the setup when labeled data is scarce. In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process. We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem. Based on these observations, we describe a simple segmentation method, which can work even if only a few training images are provided. Our approach significantly outperforms the existing alternatives on several datasets for the same amount of human supervision.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “They are also rapidly finding use in other domains such as audio [27, 37] and video [19] generation, image segmentation [4, 54] and language translation [34].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “7.1.2 Semantic Segmentation.”
Diffusion Models for Medical Anomaly Detection (2022, methodology): “In the flow of this success, they were also applied on image-to-image translation [23,7], segmentation [4], reconstruction [22] and registration[12].”
Diffusion models as plug-and-play priors (2022, methodology): “This aligns with the observations made in [3], where the intermediate representations of the denoising U-Net for different t values were used to predict attributes with varying granularity.”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, methodology): “In addition, the diffusion models have been applied to numerous vision tasks, e.g ., image segmentation [3, 4, 8, 26, 58, 67], object detection [7], and monocular depth estimation [14, 53].”
Emergent Correspondence from Image Diffusion (2023, background): “Some recent works [5, 90, 96] fine-tune either the U-Net or the attached head for dense prediction tasks and yield better performance.”
TabDDPM: Modelling Tabular Data with Diffusion Models (2022, background): “…the advantages of diffusion models are successfully exploited in applications, such as colorization (Song et al., 2021), in-painting (Song et al., 2021), segmentation (Baranchuk et al., 2021), super-resolution (Saharia et al., 2021; Li et al., 2021), semantic editing (Meng et al., 2021) and others.”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “Diffusion models have been applied to semantic segmentation, directly [1, 64, 27] or indirectly [4, 2].”
Implicit Diffusion Models for Continuous Super-Resolution (2023, positive): “While impressive PSNR results have been shown, they often suffer from duller edges and over-smoothing details in perceptual outputs.”
Diffusion Models for Implicit Image Segmentation Ensembles (2021, methodology): “Recently, one approach to perform semantic segmentation with a diffusion model was proposed by (Baranchuk et al., 2022).”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “Examples include tasks like semantic image segmentation [3, 10, 36, 37, 54, 66], optical flow estimation [14, 32, 63], human motion understanding [19, 29, 38, 67], and other dense prediction tasks [3, 70].”
A Survey on Generative Diffusion Models (2022, methodology): “It typically requires an additional classiﬁer with a UNet Encoder architecture to generate condition gradients for speciﬁc labels, which can be text, categorical, binary, or extracted features [12], [27], [28], [49], [50], [51], [52], [53], [54], [55].”
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023, methodology): “They either synthesize many images and their mask labels to train a separate segmentation network [37, 85]; or directly use the generative model to perform segmentation [2].”
Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, neutral): “Inspired by Label-DDPM [3], we extract Stable Diffusion features (mid-layer U-Net features at a resolution [ 8 ⇥ 8 ⇥ 1024 ] at timestep t = 100 ), and then ﬁt a ResNet-50 classiﬁer on the extracted features and corresponding ground-truth labels.”