Conditional Random Fields as Recurrent Neural Networks

By Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, Philip H. S. Torr. 2015

2,597 citations recorded by the source corpus.

Pixel-level labelling tasks, such as semantic segmentation, play a central role in image understanding. Recent approaches have attempted to harness the capabilities of deep learning techniques for image recognition to tackle pixel-level labelling tasks. One central issue in this methodology is the limited capacity of deep learning techniques to delineate visual objects. To solve this problem, we introduce a new form of convolutional neural network that combines the strengths of Convolutional Neural Networks (CNNs) and Conditional Random Fields (CRFs)-based probabilistic graphical modelling. To this end, we formulate Conditional Random Fields with Gaussian pairwise potentials and mean-field approximate inference as Recurrent Neural Networks. This network, called CRF-RNN, is then plugged in as a part of a CNN to obtain a deep network that has desirable properties of both CNNs and CRFs. Importantly, our system fully integrates CRF modelling with CNNs, making it possible to train the whole deep network end-to-end with the usual back-propagation algorithm, avoiding offline post-processing methods for object delineation. We apply the proposed method to the problem of semantic image segmentation, obtaining top results on the challenging Pascal VOC 2012 segmentation benchmark.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Fully convolutional networks for semantic segmentation (2014, methodology): “The joint CRFasRNN [40] model is an end-to-end integration of the CRF for further improvement.”
CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “Conditional random field (CRF) [4, 43], Markov random field (MRF) [25] are also utilized to capture long-range dependencies for semantic segmentation.”
Non-local Neural Networks (2017, background): “The iterative mean-field inference of CRF can be turned into a recurrent network and trained [56, 42, 8, 18, 34].”
UPSNet: A Unified Panoptic Segmentation Network (2019, background): “Recent semantic segmentation methods that exploit deep convolutional feature extraction mainly approach this problem from either multi-scale feature aggregation [40, 26, 5, 14], or end-to-end structured prediction [2, 41, 1, 25, 4] perspectives.”
Fully Convolutional Instance-Aware Semantic Segmentation (2016, background): “Recent works in [3, 43, 37, 24] integrated FCNs with conditional random fields (CRFs).”
Recurrent Instance Segmentation (2015, methodology): “Another extension consists in integrating the CRF module as a layer in the end-to-end model, such as in [14].”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, background): “In our case, this network is FCN8s [33] with a CRF whose inference is unrolled as an RNN and trained end-to-end, as described in [53] and [2].”
InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “of the methods utilize a conditional random field (CRF) model on top of an FCN [10, 36], or incorporate CRF-based mechanisms directly into a network architecture [39, 56, 45].”
The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, methodology): “Methods that leverage a CRF for regularization [9, 40, 48, 81] tend to over smooth small objects, c.”
Instance-Aware Semantic Segmentation via Multi-task Network Cascades (2015, background): “Nevertheless, FCNs [23] and improvements [5, 24, 6, 31] are designed to predict a category label for each pixel, but are unaware of individual object instances.”
COCO-Stuff: Thing and Stuff Classes in Context (2016, background): “Furthermore, while PASCAL Context has a seemingly impressive number of 540 classes, in practice most classes are too rare to be usable: [38, 18, 37, 17, 62, 5] only use the 60 most frequent classes, out of which only 28 are stuff classes (Table 1).”
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation (2016, methodology): “Interestingly, in [38], RNN have been introduced to approximate mean-field iterations of CRF optimization, allowing for an end-to-end training of both the FCN and the RNN.”
Discriminative Embeddings of Latent Variable Models for Structured Data (2016, neutral): “In these existing messages learning work (Hershey et al., 2014; Zheng et al., 2015; Lin et al., 2015), the learning task is still estimating the messages represented graphical models with designed function forms, e.”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “This direction has been extended by several follow-up papers [17], [40], [58], [59], [60], [61], [62], [63], [65], since the first version of our work was published [38].”
Structured Convolutional Kernel Networks for Airline Crew Scheduling (2021, methodology): “Note that unlike recurrent neural networks (RNNs) or neural networks by (Yaakoubi et al., 2020) which cannot produce initial solutions that are sufﬁciently close to being feasible, the proposed predictor incorporates local constraints in the learning process.”
Deep Interactive Object Selection (2016, methodology): “Since all existing interactive segmentation algorithms are only based on low-level features, we should also compare our algorithm to some models that understand high-level semantics, such as FCNs [17] and CRF-RNN [29].”
Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016, methodology): “Some methods move beyond per-pixel losses by penalizing image gradients [6], framing CRF inference as a recurrent layer trained jointly with the rest of the network [17], or using a CRF loss layer [18] to enforce local consistency in the output.”
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs (2014, methodology): “In Section 4.2, we pursue a novel alternative direction based on coupling the recognition capacity of DCNNs and the fine-grained localization accuracy of fully connected CRFs and show that it is remarkably successful in addressing the localization challenge, producing accurate semantic segmentation ”
Multi-Scale Context Aggregation by Dilated Convolutions (2015, methodology): “Bottom: the context module increases accuracy when plugged into a front-end + CRF-RNN conﬁguration (Zheng et al., 2015). is both feasible and desirable.”