Fast, Exact and Multi-scale Inference for Semantic Image Segmentation with Deep Gaussian CRFs

By Siddhartha Chandra, Iasonas Kokkinos. 2016

191 citations recorded by the source corpus.

In this work we propose a structured prediction technique that combines the virtues of Gaussian Conditional Random Fields (G-CRF) with Deep Learning: (a) our structured prediction task has a unique global optimum that is obtained exactly from the solution of a linear system (b) the gradients of our model parameters are analytically computed using closed form expressions, in contrast to the memory-demanding contemporary deep structured prediction approaches [1, 2] that rely on back-propagation-through-time, (c) our pairwise terms do not have to be simple hand-crafted expressions, as in the line of works building on the DenseCRF [1, 3], but can rather be ‘discovered’ from data through deep architectures, and (d) out system can trained in an end-to-end manner. Building on standard tools from numerical analysis we develop very efficient algorithms for inference and learning, as well as a customized technique adapted to the semantic segmentation task. This efficiency allows us to explore more sophisticated architectures for structured prediction in deep learning: we introduce multi-resolution architectures to couple information across scales in a joint optimization framework, yielding systematic improvements. We demonstrate the utility of our approach on the challenging VOC PASCAL 2012 image segmentation benchmark, showing substantial improvements over strong baselines. We make all of our code and experiments available at https://github.com/siddharthachandra/gcrf.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “BiSeNet [35] was designed for real-time semantic segmentation.”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, background): “Another fruitful direction pursued by [40], [66] is to learn the pairwise terms of a CRF via a DCNN, significantly improving performance at the cost of heavier computation.”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, background): “, image pyramid) [18, 16, 58, 44, 11, 9] or those that adopt probabilistic graphical models (such as DenseCRF [37] with efficient inference algorithm [2]) [8, 4, 82, 44, 48, 55, 63, 34, 72, 6, 7, 9].”
Rethinking Atrous Convolution for Semantic Image Segmentation (2017, background): “Recently, [35] proposes to learn a general and sparse high-dimensional convolution (bilateral convolution), and [70, 7] combine Gaussian Conditional Random Fields and DCNNs for semantic segmentation.”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, methodology): “There are certain straightforward directions for future work: (i) considering more tasks, such as symmetry, human landmarks, texture segmentation, or any other of the tasks indicated in the introduction (ii) using deeper architectures, such as ResNets [40] (iii) combining the dense labelling results”
Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network (2017, methodology): “FCN-8s-heavy [29] 67.2 TTI zoomout v2 [26] 69.6 MSRA BoxSup [9] 71.0 DeepLab-MSc-CRF-LargeFOV [6] 71.6 Oxford TVG CRF RNN COCO [37] 74.7 CUHK DPN COCO [24] 77.5 Oxford TVG HO CRF [2] 77.9 CASIA IVA OASeg [33] 78.3 Adelaide VeryDeep FCN VOC [34] 79.1 LRR 4x ResNet COCO [12] 79.3 Deeplabv2-CRF [7] 79.”
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation (2016, neutral): “7 CSupelec-Res101 [4] 92.”
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes (2016, neutral): “Most commonly, conditional random ﬁelds (CRFs) [30] are applied on the network output [9, 8, 12, 34, 6].”
Dense and Low-Rank Gaussian CRFs Using Deep Embeddings (2017, methodology): “Structured prediction combined with deep learning has delivered excellent results on a variety of computer vision benchmarks [2, 5, 7, 8, 33, 34, 40].”
Discovering Class-Specific Pixels for Weakly-Supervised Semantic Segmentation (2017, background): “Convolutional Neural Networks (CNNs) are extremely successful in solving structured output prediction tasks such as semantic segmentation [4, 5, 19, 35], where the goal is to assign a semantic class label to each pixel.”
On the Robustness of Semantic Segmentation Models to Adversarial Attacks (2017, background): “This has also been witnessed by the fact that state-of-the-art semantic segmentation models are typically based on standard image classification architectures [41, 62, 33], extended by additional components such as dilated convolutions [14, 69], specialised pooling [15, 71], skip-connections [47, 7]”
Pixel-Adaptive Convolutional Neural Networks (2019, background): “One class of techniques make traditional image-adaptive filters, such as bilateral filters [2, 41] and guided image filters [18] differentiable, and use them as layers inside a CNN [24, 28, 51, 11, 9, 21, 29, 8, 13, 30, 43, 45].”
Survey on semantic segmentation using deep learning techniques (2019, background): “Chandra et al. [158] proposed a Gaussian Conditional Random Field (G-CRF) module using a quadratic energy function that captures unary and pairwise interactions.”
PSANet: Point-wise Spatial Attention Network for Scene Parsing (2018, methodology): “Different from these extensions, conditional random field (CRF) [4,46,2,3] and Markov random field (MRF) [25] are also utilized.”