Monocular Object Instance Segmentation and Depth Ordering with CNNs

By Ziyu Zhang, Alexander Gerhard Schwing, Sanja Fidler, Raquel Urtasun. 2015

164 citations recorded by the source corpus.

In this paper we tackle the problem of instance-level segmentation and depth ordering from a single monocular image. Towards this goal, we take advantage of convolutional neural nets and train them to directly predict instance-level segmentations where the instance ID encodes the depth ordering within image patches. To provide a coherent single explanation of an image we develop a Markov random field which takes as input the predictions of convolutional neural nets applied at overlapping patches of different resolutions, as well as the output of a connected component algorithm. It aims to predict accurate instance-level segmentation and depth ordering. We demonstrate the effectiveness of our approach on the challenging KITTI benchmark and show good performance on both tasks.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Recurrent Instance Segmentation (2015, background): “In this section we summarize the main developments in these areas.”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, background): “The works of [50] and [51] order instances by depth to break this symmetry.”
Shape-aware Instance Segmentation (2016, background): “By contrast, [33] proposed to identify the individual instances based on their depth ordering.”
Deep Watershed Transform for Instance Segmentation (2016, methodology): “Deep structured models: [30, 31] ﬁrst use CNNs to perform local instance disambiguation and labelling, followed by a global CRF to achieve instance label consistency.”
InstanceCut: From Edges to Instances with MultiCut (2016, background): “Another work, which focuses on instance segmentation of cars [55, 54] employs a conditional random field that reasons about instances using multiple overlapping outputs of an FCN.”
The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, methodology): “Following this line, [9, 37, 40, 63, 81] propose different approaches that combine the strengths of CNNs and Conditional Random Fields (CRFs).”
Hybrid Task Cascade for Instance Segmentation (2019, neutral): “[46, 45] propose to predict instance labels based on local patches and integrate the local results with an MRF.”
Path Aggregation Network for Instance Segmentation (2018, methodology): “A graphical model was used in [66, 65] to infer the order of instances.”
Proposal-Free Network for Instance-Level Object Segmentation (2015, positive): “In this work, we follow some of the recent works [14] [3] [39] and attempt to solve a more challenging task, instancelevel object segmentation, which predicts the segmentation mask for each instance of each category.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, background): “We note that there are several instance segmentation works [77, 68, 76, 2, 48, 35, 56, 20, 18, 44, 37, 30, 51, 54, 6], which could be potentially extended to bottom-up panoptic segmentation.”
Pixel-Level Encoding and Depth Layering for Instance-Level Semantic Labeling (2016, methodology): “We clearly outperform all existing works on KITTI (Best [34]/[35]), c.”
Visual question answering: Datasets, algorithms, and future challenges (2016, background): “Instance segmentation further builds upon localization by differentiating between separate instances of the same semantic class [14, 15, 16].”
Meta R-CNN: Towards General Solver for Instance-Level Low-Shot Learning (2019, methodology): “Proposalbased methods [30, 31, 7, 6] predict object masks based on the generated region proposals while image-based methods [47, 48, 44, 2] produce a pixel-level segmentation map over the image to identify object instance.”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, methodology): “[78] exploit depth ordering within an image patch.”
Instance-Level Segmentation for Autonomous Driving with Deep Densely Connected MRFs (2015, methodology): “Our experimental evaluation shows significant improvements over [32] on the challenging KITTI benchmark [8].”
Reversible Recursive Instance-Level Object Segmentation (2015, background): “Recently, beyond the traditional object detection [7][24][5][14][4][25] and semantic segmentation tasks [1][18][2][32][16], instance-level object segmentation has attracted much attention [10][11][17][26][31][15].”
Semantic Instance Segmentation with a Discriminative Loss Function (2017, methodology): “Another line of research is to train a network to transform the image into a representation that is clustered into individual instances with a post-processing step [34, 46, 37, 22].”
DeeperLab: Single-Shot Image Parser (2019, methodology): “We note that Covering has been used in several instance segmentation works [92, 74, 76, 93, 78].”
End-to-End Instance Segmentation and Counting with Recurrent Attention (2016, methodology): “Recent work on instance segmentation [21, 27, 26] formulates complex graphical models, which results in a complex and time-consuming pipeline.”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, methodology): “We define the problem of instance segmentation as labelling every pixel in an image with both its object class and an instance identifier [16,17,18].”