Semantic Segmentation with Boundary Neural Fields

By Gedas Bertasius, Jianbo Shi, Lorenzo Torresani. 2015

211 citations recorded by the source corpus.

The state-of-the-art in semantic segmentation is currently represented by fully convolutional networks (FCNs). However, FCNs use large receptive fields and many pooling layers, both of which cause blurring and low spatial resolution in the deep layers. As a result FCNs tend to produce segmentations that are poorly localized around object boundaries. Prior work has attempted to address this issue in post-processing steps, for example using a color-based CRF on top of the FCN predictions. However, these approaches require additional parameters and low-level features that are difficult to tune and integrate into the original network architecture. Additionally, most CRFs use colorbased pixel affinities, which are not well suited for semantic segmentation and lead to spatially disjoint predictions. To overcome these problems, we introduce a Boundary Neural Field (BNF), which is a global energy model integrating FCN predictions with boundary cues. The boundary information is used to enhance semantic segment coherence and to improve object localization. Specifically, we first show that the convolutional filters of semantic FCNs provide good features for boundary detection. We then employ the predicted boundaries to define pairwise potentials in our energy. Finally, we show that our energy decomposes semantic segmentation into multiple binary problems, which can be relaxed for efficient global optimization. We report extensive experiments demonstrating that minimization of our global boundary-based energy yields results superior to prior globalization methods, both quantitatively as well as qualitatively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “Recent advances in deep learning have made this class of methods especially efﬁcient, since they automatically obtain rich feature representation for classiﬁcation [19, 29, 46, 6, 7, 51, 8].”
One-Shot Video Object Segmentation (2016, methodology): “Image segmentation has also been taken over by CNNs recently [29, 22, 53, 3, 4], with deep architectures pre-trained on the weakly related task of image classification on ImageNet [44].”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, background): “In recent years, convolutional networks have obtained remarkable results in still-image object detection [16, 33, 14, 15], and segmentation [27, 45, 8, 4].”
Rethinking Semantic Segmentation: A Prototype View (2022, background): “Since it was proposed, numerous efforts have been devoted to improving FCN, by, for example, enlarging the receptive field [15, 16, 25, 124, 128, 135]; strengthening context cues [4, 43, 47, 48, 56, 57, 70, 75, 77, 81, 90, 126, 128, 129, 132, 138, 141]; leveraging boundary information [6, 14, 27, 66”
Weakly Supervised Object Boundaries (2015, background): “More recently [4, 5] consider mixing a semantic labelling convnet with a generic boundary detection convnet, to obtain class specific boundaries.”
ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data (2019, background): “The second is the common boundary between the segmentation masks that is known to improve performance for semantic segmentation (Bertasius et al., 2015; Marmanis et al., 2018).”
Adaptive Affinity Field for Semantic Segmentation (2018, background): “Besides, low-level cues, such as affinity [25,26] and contour [27,28] have also been leveraged to encode image structures.”
Salient Object Detection With Pyramid Attention and Salient Edges (2019, methodology): “Inspired by recent advances in semantic segmentation [4, 6], we propose to equip saliency models with a salient-edge detection module, specially designed to detect the salient object boundaries.”
Survey on semantic segmentation using deep learning techniques (2019, methodology): “…have been proposed to overcome these issues and can be used as alternate to CRFs. Table 11 shows network models alternate to CRFs. Bertasius et al. [173] proposed a FCN architecture named Boundary Neural Field (BNF) to predict semantic boundaries and produce semantic segmentation maps using global…”
Improving Semantic Segmentation via Decoupled Body and Edge Supervision (2020, methodology): “For the second problem, prior arts manage to embed the low-level features that contain boundary and edge information into high-level features [18,19,20,21] or directly reﬁne the outputs [22].”
Learning to predict crisp boundaries (2018, background): “Nowadays, it is expected to localize semantically meaningful objects boundaries, which play a fundamental and signiﬁcant role in many computer vision tasks such as image segmentation [4,5,6,7] and optical ﬂow [8,9].”
Convolutional Oriented Boundaries: From Image Segmentation to High-Level Tasks (2016, methodology): “[4], [32] show that results can be significantly improved when using deep-learning based semantic segmenters and contour”
Improving Semantic Segmentation via Video Propagation and Label Relaxation (2018, background): “There is also literature focusing on structure modeling to obtain better boundary localization, such as affinity field [21], random walk [5], relaxation labelling [37], boundary neural fields [4], etc.”
SegFix: Model-Agnostic Boundary Refinement for Segmentation (2020, background): “In general, we only need to train a single unified SegFix model to improve the boundary quality of various segmentation models across different datasets, thus SegFix is much more training friendly (and saves a lot of energy consumption) compared to the previous methods [5,52,17,41,40,30] that requir”
Segmentation-Aware Convolutional Networks Using Local Attention Masks (2017, methodology): “, using edges found in the image [8, 4], or using a dense conditional random field (CRF) [28, 9, 58].”
Webly Supervised Semantic Segmentation (2017, methodology): “Recent breakthroughs [1] [2] [3] [4] [5] [6] in semantic segmentation are mainly due to the fully supervised algorithms that apply Convolutional Neural Networks (CNNs) on datasets that contain images and their pixel-wise annotations , e.g., PASCAL VOC [7] and Microsoft COCO [8].”