Higher Order Conditional Random Fields in Deep Neural Networks

By Anurag Arnab, Sadeep Jayasumana, Shuai Zheng, Philip H. S. Torr. 2015

232 citations recorded by the source corpus.

We address the problem of semantic segmentation using deep learning. Most segmentation systems include a Conditional Random Field (CRF) to produce a structured output that is consistent with the image's visual features. Recent deep learning approaches have incorporated CRFs into Convolutional Neural Networks (CNNs), with some even training the CRF end-to-end with the rest of the network. However, these approaches have not employed higher order potentials, which have previously been shown to significantly improve segmentation performance. In this paper, we demonstrate that two types of higher order potential, based on object detections and superpixels, can be included in a CRF embedded within a deep network. We design these higher order potentials to allow inference with the differentiable mean field algorithm. As a result, all the parameters of our richer CRF model can be learned end-to-end with our pixelwise CNN classifier. We achieve state-of-the-art segmentation performance on the PASCAL VOC benchmark with these trainable higher order potentials.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

UPSNet: A Unified Panoptic Segmentation Network (2019, background): “Recent semantic segmentation methods that exploit deep convolutional feature extraction mainly approach this problem from either multi-scale feature aggregation [39, 26, 5, 14], or end-to-end structured prediction [2, 40, 1, 25, 4] perspectives.”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, methodology): “However, inference of this graphical model was not performed end-toend as shown to be possible in [53, 2, 4, 29].”
The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, methodology): “The best performing baseline for pixel-level semantic segmentation obtains an IoU score of 67.1%, whereas the best current methods on PASCAL VOC and KITTI reach IoU levels of 77.9% [3] and 72.5% [73], respectively.”
COCO-Stuff: Thing and Stuff Classes in Context (2016, background): “Furthermore, while PASCAL Context has a seemingly impressive number of 540 classes, in practice most classes are too rare to be usable: [38, 18, 37, 17, 62, 5] only use the 60 most frequent classes, out of which only 28 are stuff classes (Table 1).”
Pyramid Scene Parsing Network (2016, methodology): “Following methods [25, 41, 1] refined networks via end-to-end modeling.”
Conditional Random Fields as Recurrent Neural Networks (2015, background): “Comparison results on VOC 12 reduced-validation set Results on Pascal-context validation set Method FCN [3] DeepLab [1] CRF-RNN Mean IOU 68.”
Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network (2017, methodology): “…v2 [26] 69.6 MSRA BoxSup [9] 71.0 DeepLab-MSc-CRF-LargeFOV [6] 71.6 Oxford TVG CRF RNN COCO [37] 74.7 CUHK DPN COCO [24] 77.5 Oxford TVG HO CRF [2] 77.9 CASIA IVA OASeg [33] 78.3 Adelaide VeryDeep FCN VOC [34] 79.1 LRR 4x ResNet COCO [12] 79.3 Deeplabv2-CRF [7] 79.7 CentraleSupelec Deep…”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Early methods focused on modeling contextual relationships [2,49,103].”
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation (2016, neutral): “5 HO-CRF [1] 41.”
Context Encoding for Semantic Segmentation (2018, neutral): “5 HO CRF [2] 41.”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, methodology): “[4, 5] apply higher-order Conditional Random Fields (CRFs) to refine the mask results.”
Learning Affinity via Spatial Propagation Networks (2017, methodology): “Some methods use CFR as a post-processing module [5], while others integrate it as a jointly-trained part [36, 17, 24, 1].”
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing (2020, methodology): “Early techniques for modeling contextual relationships for scene parsing involve the conditional random ﬁelds (CRFs) [25, 49, 1, 67].”
Bottom-up Instance Segmentation using Deep Higher-Order CRFs (2016, methodology): “In our work, we extend the end-to-end trainable, higher-order, object detection potentials proposed by [2] to the task of instance segmentation.”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, methodology): “The main difference of our model to [16] is that our network is based on the PSPNet architecture using ResNet-101, whilst [16] used the network of [66] based on VGG [2].”