A Probabilistic U-Net for Segmentation of Ambiguous Images

By Simon A. A. Kohl, Bernardino Romera-Paredes, Clemens Meyer, Jeffrey De Fauw, J. Ledsam, Klaus Hermann Maier-Hein, S. M. Ali Eslami, Danilo Jimenez Rezende, Olaf Ronneberger. Neural Information Processing Systems, 2018

731 citations recorded by the source corpus.

Many real-world vision problems suffer from inherent ambiguities. In clinical applications for example, it might not be clear from a CT scan alone which particular region is cancer tissue. Therefore a group of graders typically produces a set of diverse but plausible segmentations. We consider the task of learning a distribution over segmentations given an input. To this end we propose a generative segmentation model based on a combination of a U-Net with a conditional variational autoencoder that is capable of efficiently producing an unlimited number of plausible hypotheses. We show on a lung abnormalities segmentation task and on a Cityscapes segmentation task that our model reproduces the possible segmentation variants as well as the frequencies with which they occur, doing so significantly better than published approaches. These models could have a high impact in real-world applications, such as being used as clinical decision-making algorithms accounting for multiple plausible semantic segmentation hypotheses to provide possible diagnoses and recommend further actions to resolve the present ambiguities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/df28b3c8535e7e36b3f3f824a5145aa49f791e6d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Accurate brain age prediction with lightweight deep neural networks (2019, background): “The recent development of deep learning pushes prediction accuracy beyond human performance in some scenarios, and is able to assist clinical diagnosis/treatment decisions (De Fauw et al., 2018; Kohl et al., 2018; LeCun et al., 2015).”
Sketch-a-Segmenter: Sketch-Based Photo Segmenter Generation (2020, methodology): “In order to keep precise localization information, feature maps from the encoder are combined with their corresponding part in the decoder for prediction [25], [26].”
A Topological Loss Function for Deep-Learning based Image Segmentation using Persistent Homology (2019, background): “which are coherent in a global sense [5].”
Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation (2019, methodology): “[31] proposed a Probabilistic U-Net that combined an FCN with a conditional variance autoencoder to provide multiple segmentation hypotheses for ambiguous images.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “[431] devised a generative segmentation model based on the combination of a U-Net with a conditional VAE which is capable of generating large number of plausible hypotheses.”
Spatial proteomics: a powerful discovery tool for cell biology (2019, background): “…in this context include convolutional networks (such as U-Net 136 ), the advent of generative models capable of the segmentation of ambiguous images 137 and in silico synthesis of realistic fluorescent labels from label-free images, such as prediction of labels representing cell state (dead versus…”
Direct Uncertainty Prediction for Medical Second Opinions (2018, methodology): “Related techniques have also been explored in semantic segmentation (Gurari et al., 2018; Kohl et al., 2018).”
Surgical data science – from concepts toward clinical translation (2020, background): “This could potentially be addressed with fuzzy output/references as well as with probabilistic methods capable of representing multiple plausible solutions in the output, as suggested by some early work on the topic (Kohl et al., 2018; Adler et al., 2019; Trofimova et al., 2020).”
A Review on Deep Learning Techniques for Video Prediction (2020, neutral): “[15] 2020 [56], [163], [219] cED [121], [122], [220], [221] RGB SS,D,M X CE, `δ, Ld, Lc, Lp R ? ? ? X 7”
MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer (2023, background): “GED is a commonly used metric to measures the agreement between predictions and the ground truth distribution of segmentation by comparing their distributions(Kohl et al. 2018).”
AI in Medical Imaging Informatics: Current Challenges and Future Directions (2020, positive): “The limitations of such approaches for medical image segmentation were quickly realised and led to the discovery of U-Net [104], which is even today one of the most successful architectures for medical image segmentation.”
Deep Learning in Medical Imaging (2019, background): “50 51-53 Since these studies have not yet surpassed the segmentation performance of supervised learning, it is considered future value as a technology that can overcome the severe imbalances in medical imaging.”
Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models (2020, methodology): “We follow the settings in [31, 48] that crops the original images into 128 × 128 patches centered on a lesion for which at least one radiologist has annotated.”
Deep learning in histopathology: the path to the clinic (2021, background): “An alternative approach is the development of CNN techniques that yield, in addition to the network output, a score that expresses the certainty of the CNN for that specific output 125,126 , which would offer CNNs the ability to essentially state ‘I do not know’.”
Applying and improving AlphaFold at CASP14 (2021, methodology): “The initial MSA contained very few alignments (19) and gave a low confidence prediction.”
Medical Image Segmentation Review: The Success of U-Net (2022, neutral): “Taking this into consideration, Kohl et al. learn a distribution over segmentations from an ambiguous input to produce an unlimited number of possible segmentations instead of just providing the most likely hypothesis [50].”
UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders (2020, methodology): “Typical using of CVAE involves multiple versions of ground truth Y [32] to produce informative z ∈ R , with each position in z represents possible labeling variants or factors that may cause diverse saliency annotations.”