Simultaneous Detection and Segmentation

By Bharath Hariharan, Pablo Arbeláez, Ross B. Girshick, Jitendra Malik. 2014

1,345 citations recorded by the source corpus.

We aim to detect all instances of a category in an image and, for each instance, mark the pixels that belong to it. We call this task Simultaneous Detection and Segmentation (SDS). Unlike classical bounding box detection, SDS requires a segmentation and not just a box. Unlike classical semantic segmentation, we require individual object instances. We build on recent work that uses convolutional neural networks to classify category-independent region proposals (R-CNN [16]), introducing a novel architecture tailored for SDS. We then use category-specific, top-down figure-ground predictions to refine our bottom-up proposals. We show a 7 point boost (16% relative) over our baselines on SDS, a 5 point boost (10% relative) over state-of-the-art on semantic segmentation, and state-of-the-art performance in object detection. Finally, we provide diagnostic tools that unpack performance and provide directions for future work.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Object Detection via a Multi-region and Semantic Segmentation-Aware CNN Model (2015, neutral): “The motivation for this comes by the close connection between segmentation and detection and by the fact that segmentation related cues are empirically known to often help object detection [5, 11, 23].”
Convolutional neural networks at constrained time cost (2014, background): “Deep CNNs not only have been continuously advancing the image classification accuracy [14, 21, 24, 1, 9, 22, 23], but also play as generic feature extractors for various recognition tasks such as object detection [6, 9], semantic segmentation [6, 8], and image retrieval [14, 19].”
Fully convolutional networks for semantic segmentation (2014, methodology): “2 Sampling in patchwise training can correct class imbalance [27, 7, 2] and mitigate the spatial correlation of dense patches [28, 15].”
Mask R-CNN (2017, methodology): “Earlier methods [13, 15, 16, 9] resorted to bottom-up segments [38, 2].”
Webly Supervised Learning of Convolutional Networks (2015, methodology): “To demonstrate the effectiveness of our representation, we do two experiments: (a) First, we use our ﬁnal trained network using both Google and Flickr images to test on VOC 2007 and 2012 dataset.”
Colorful Image Colorization (2016, methodology): “Methods for performing pixel-prediction have been explored for the tasks of semantic segmentation [13], segmentation [20], and simultaneous detection and segmentation [21].”
UPSNet: A Unified Panoptic Segmentation Network (2019, background): “Beginning with the introduction of region-based CNN (R-CNN) [12], many early deep learning approaches to instance segmentation attacked the problem by casting the solution to instance segmentation as a two stage approach where a number of segment proposals are made, which is then followed by a votin”
Panoptic Segmentation (2018, background): “The standard metric for instance segmentation is Average Precision (AP) [25, 13].”
Recurrent Neural Networks for Semantic Instance Segmentation (2017, methodology): “[8] present a post-processing stage for occlusion handling to refine the predictions from [17].”
Fully Convolutional Instance-Aware Semantic Segmentation (2016, methodology): “Accuracy is evaluated by mean average precision, mAP [15], at mask-level IoU (intersection-over-union) thresholds at 0.”
Recurrent Instance Segmentation (2015, background): “Keywords: Instance segmentation, recurrent neural nets, deep learning”
Learning to decompose for object detection and instance segmentation (2015, background): “An instance map is correct if its segmentation overlaps with the segmentation of a ground truth instance by more that pre-defined threshold(Hariharan et al., 2014; Liang et al., 2015).”
You Only Look Once: Unified, Real-Time Object Detection (2015, neutral): “1 SDS [16] 50.”
Hypercolumns for object segmentation and fine-grained localization (2014, methodology): “Substituting hypercolumns into the pipeline of [22] improves this to 52.”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, methodology): “We do not require any post-processing, such as the patch aggregation of [32], “mask-voting” of [11], “superpixel projection” of [18, 19, 26] or spectral clustering of [28].”
Beyond Skip Connections: Top-Down Modulation for Object Detection (2016, methodology): “After the resurgence of ConvNets for image classification [8, 29], they have been successfully adopted for a variety of computer vision tasks such as object detection [16, 17, 41, 48], semantic segmentation [3, 6, 34, 35], instance segmentation [21, 22, 38], pose estimation [49, 50], depth estimatio”
Shape-aware Instance Segmentation (2016, methodology): “These proposals can be learned [13, 19, 7] or sampled by sliding windows [23, 5], and greatly facilitate the task of identifying the different instances, may they be from the same category or different ones.”
Deep Watershed Transform for Instance Segmentation (2016, methodology): “In a similar spirit, [12] selects proposals using CNN features and non-maximum suppression.”
InstanceCut: From Edges to Instances with MultiCut (2016, background): “MCG [3] and then, in the second stage, a different network classifies these proposals [13, 22, 15, 11].”