SegDiff: Image Segmentation with Diffusion Probabilistic Models

By Tomer Amit, Eliya Nachmani, Tal Shaharbany, Lior Wolf. arXiv.org, 2021

443 citations recorded by the source corpus.

Diffusion Probabilistic Methods are employed for state-of-the-art image generation. In this work, we present a method for extending such models for performing image segmentation. The method learns end-to-end, without relying on a pre-trained backbone. The information in the input image and in the current estimation of the segmentation map is merged by summing the output of two encoders. Additional encoding layers and a decoder are then used to iteratively refine the segmentation map, using a diffusion model. Since the diffusion model is probabilistic, it is applied multiple times, and the results are merged into a final segmentation map. The new method produces state-of-the-art results on the Cityscapes validation set, the Vaihingen building segmentation benchmark, and the MoNuSeg dataset.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1c1cc65908dd5ca3d0103d650de896e053e0e7d7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model (2022, methodology): “We also included SegDiff(Amit et al., 2021) and SegNet(Badrinarayanan et al., 2017), which were proposed for natural image segmentation, because of their close relationship with our method.”
UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models (2023, methodology): “References [1] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf.”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, methodology): “In addition, the diffusion models have been applied to numerous vision tasks, e.g ., image segmentation [3, 4, 8, 26, 58, 67], object detection [7], and monocular depth estimation [14, 53].”
Diffusion models in medical imaging: A comprehensive survey (2022, background): “To date, diffusion models have been found to be useful in a wide variety of areas, ranging from generative modeling tasks such as image generation [11], image super-resolution [12], image inpainting [13] to discriminative tasks such as image segmentation [14], classification [15], and anomaly detect”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “have broken the long-time dominance of generative adversarial networks (GANs) [73] in the challenging task of image synthesis [49, 90, 220, 225] and have also shown potential in a variety of domains, ranging from computer vision [2, 9, 17, 20, 91, 93, 115, 118, 137, 151, 161, 174, 200, 202, 245, 267”
Diffusion models as plug-and-play priors (2022, methodology): “Deep generative models, such as denoising diffusion probabilistic models [DDPMs; 37, 13] can capture the details of very complex distributions over high-dimensional continuous data p(x) [28, 7, 1, 36, 40, 15].”
Diffusion Models for Implicit Image Segmentation Ensembles (2021, methodology): “In (Amit et al., 2021), a similar experiment was performed on a diﬀerent data set.”
Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation (2023, unknown): “In the tables, we compare MSA with the SAM model, as well as widely-used and well-recognized medical image segmentation methods, including nnUNet [19], TransUNet [3], UNetr [15], Swin-UNetr [14], EnsDiff [35], SegDiff [1], and MedSegDiff [36], all of which are based on either convolutional neural ne”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “Diffusion models have been applied to semantic segmentation, directly [1, 64, 27] or indirectly [4, 2].”