Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials

By Philipp Krahenbuhl, Vladlen Koltun. 2011

3,724 citations recorded by the source corpus.

Most state-of-the-art techniques for multi-class image segmentation and labeling use conditional random fields defined over pixels or image regions. While region-level models often feature dense pairwise connectivity, pixel-level models are considerably larger and have only permitted sparse graph structures. In this paper, we consider fully connected CRF models defined on the complete set of pixels in an image. The resulting graphs have billions of edges, making traditional inference algorithms impractical. Our main contribution is a highly efficient approximate inference algorithm for fully connected CRF models in which the pairwise edge potentials are defined by a linear combination of Gaussian kernels. Our experiments demonstrate that dense connectivity at the pixel level substantially improves segmentation and labeling accuracy.

Subfields: Biomedical Engineering, Computational Mechanics, Computer Networks and Communications, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computational Mechanics

In Computer Networks and Communications

In Model Architectures

Citations in the corpus

Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, background): “This CRF contains the densely-connected pairwise potentials described in [23] and is formulated as a recurrent neural network as in [53].”
InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “Note, that we also use the CNN-CRF frameworks [56, 10] with dense CRF [31], since dense CRF’s output can also be treated as the log-probability scores ai,l.”
COCO-Stuff: Thing and Stuff Classes in Context (2016, background): “Boundary complexity is defined as the ratio of pixels that have any neighboring pixel with a different semantic label (as in the boundary evaluation in [8, 32, 33]).”
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation (2016, methodology): “More recently, fully connected CRFs [15] have been used to include structural properties of the output of FCNs [5].”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “Substantial improvements have been achieved by incorporating richer information from context [45] and structured prediction techniques [22], [26], [27], [46], but the performance of these systems has always been compromised by the limited expressive power of the features.”
Getting to 99% Accuracy in Interactive Segmentation (2020, neutral): “[9] ResNet-101 Deeplab-v2 [27] 1:8 CRF [28]”
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs (2014, methodology): “This forced them to use learned upsampling layers, signiﬁcantly increasing the complexity and training time of their system: Fine-tuning our network on PASCAL VOC 2012 takes about 10 hours, while they report a training time of several days (both timings on a modern GPU).”
Multi-Scale Context Aggregation by Dilated Convolutions (2015, methodology): “The context module with the dense CRF, using the original implementation of Kr¨ahenb¨uhl & Koltun (2011), performs on par with the very recent CRF-RNN (Zheng et al., 2015).”
Conditional Random Fields as Recurrent Neural Networks (2015, methodology): “The complete system, therefore, unifies strengths of both CNNs and CRFs and is trainable end-to-end using the back-propagation algorithm [32] and the Stochastic Gradient Descent (SGD) procedure.”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, methodology): “In this subsection, we evaluate the segmentation accuracy with the trimap experiment [36, 37, 9] to quantify the accuracy of the proposed decoder module near object boundaries.”
Learning Deconvolution Network for Semantic Segmentation (2015, methodology): “Finally, we apply the fully-connected CRF [16] to the output maps for the final pixel-wise labeling, where unary potential are obtained from the pixel-wise class conditional probability maps.”
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation (2015, methodology): “Then we apply a common post-processing method [24] (dense CRF) to sharpen the object boundary for generating the ﬁnal high-resolution prediction.”
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation (2015, background): “The impact of dense CRF [62] post-processing can be”
Fully Connected Deep Structured Networks (2015, background): “Densely connected models were previously considered by [17, 33, 34, 18] and shown to yield impressive results for the image segmentation task.”
Semantic Image Segmentation via Deep Parsing Network (2015, methodology): “[16] attained accurate segmentation boundary by inferring on a fullyconnected graph.”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “We leave for future work the inclusion of additional taskspecific convolutional filters [18, 42, 43, 49] or structured prediction operations [8, 10, 30, 37, 57].”
BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation (2015, methodology): “A CRF [6, 18] for post-processing is also used.”