Pixel-Level Encoding and Depth Layering for Instance-Level Semantic Labeling

By Jonas Uhrig, Marius Cordts, Uwe Franke, Thomas Brox. 2016

180 citations recorded by the source corpus.

Recent approaches for instance-aware semantic labeling have augmented convolutional neural networks (CNNs) with complex multi-task architectures or computationally expensive graphical models. We present a method that leverages a fully convolutional network (FCN) to predict semantic labels, depth and an instance-based encoding using each pixel’s direction towards its corresponding instance center. Subsequently, we apply low-level computer vision techniques to generate state-of-the-art instance segmentation on the street scene datasets KITTI and Cityscapes. Our approach outperforms existing works by a large margin and can additionally predict absolute distances of individual instances from a monocular image as well as a pixel-level semantic labeling.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Environmental Engineering

Citations in the corpus

Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, methodology): “5, our method sets a new state-of-theart on Cityscapes, surpassing concurrent work [20] and the best previous published work [42] by significant margins.”
Shape-aware Instance Segmentation (2016, methodology): “To overcome this, the recent work of [30] uses an FCN to jointly predict depth, semantics and an instance-based direction encoding.”
Deep Watershed Transform for Instance Segmentation (2016, methodology): “Template matching: [28] extracts image features using CNNs to assign a sector label to each pixel in an object instance, which corresponds to one of eight discretized radial bins around the object’s visible center.”
InstanceCut: From Edges to Instances with MultiCut (2016, background): “In [48] the authors propose a template matching scheme for instance-aware segmentation based on three modalities: predicted semantic segmentation, depth estimation, and perpixel direction estimation with respect to the center of the corresponding instance.”
Proposal-Free Network for Instance-Level Object Segmentation (2015, neutral): “[35] proposed to produce a semantic segmentation and instance-aware angle map that encoding the instance centroids.”
Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics (2017, methodology): “[41] learn semantic and instance segmentations under a classification setting.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “Finally, our model employs the parallel multi-head prediction framework similar to [68, 36, 55].”
Bridging Category-level and Instance-level Semantic Image Segmentation (2016, background): “As pointed out by Uhrig et al. [13], PFN has a complex architecture with many interleaved building blocks, which makes training very complex.”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, methodology): “[70] further train an FCN to predict instance center direction.”
Semantic Instance Segmentation with a Discriminative Loss Function (2017, background): “Another line of research is to train a network to transform the image into a representation that is clustered into individual instances with a post-processing step [34, 46, 37, 22].”
Understanding Convolution for Semantic Segmentation (2017, methodology): “Recently, [18, 37] predict the label map by applying a classiﬁer on a per-pixel basis, as it is more statistically efﬁcient.”
DeeperLab: Single-Shot Image Parser (2019, methodology): “On the other hand, the bottom-up approaches [73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84] generally adopt a two-stage processing: pixel-level predictions produced by the segmentation module are clustered together to form the instancelevel predictions.”
Joint Graph Decomposition & Node Labeling: Problem, Algorithms, Applications (2016, methodology): “Figure 2: Instance center directions with color coding from [15].”
Multimodal End-to-End Autonomous Driving (2019, background): “based [10], [11], [12], [13], [14], [15], and multimodal [16], [17], [18], [19], [20], [21], [22], [23]); object tracking (imagebased [24], [25], [26], [27], [28], [29], [30], multimodal [31]); traffic sign recognition [32]; semantic segmentation (by class/category [33], [34], [35] and object instan”
Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey (2020, methodology): “The authors of [169], [170] has introduced direction features to predict different instances of a particular object.”