segDeepM: Exploiting segmentation and context in deep neural networks for object detection

By Yukun Zhu, Raquel Urtasun, Ruslan Salakhutdinov, Sanja Fidler. 2015

161 citations recorded by the source corpus.

In this paper, we propose an approach that exploits object segmentation in order to improve the accuracy of object detection. We frame the problem as inference in a Markov Random Field, in which each detection hypothesis scores object appearance as well as contextual information using Convolutional Neural Networks, and allows the hypothesis to choose and score a segment out of a large pool of accurate object segmentation proposals. This enables the detector to incorporate additional evidence when it is available and thus results in more accurate detections. Our experiments show an improvement of 4.1% in mAP over the R-CNN baseline on PASCAL VOC 2010, and 3.4% over the current state-of-the-art, demonstrating the power of our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Object Detection via a Multi-region and Semantic Segmentation-Aware CNN Model (2015, neutral): “Also, in the contemporaneous with us work of Zhu et al. [31], the authors extract contextual features from an additional region and utilize a MRF inference framework to exploit object segmentation proposals (obtained through parametric min-cuts).”
Fast R-CNN (2015, methodology): “These methods ( e.g ., [9, 10, 18, 24]) train networks in multi-stage pipelines, which makes training difﬁcult and time consuming.”
Learning to Segment Object Candidates (2015, background): “Since then, most stateof-the-art object detectors for both the PASCAL VOC [7] and ImageNet [5] datasets rely on object proposals as a first preprocessing step [10, 15, 33].”
A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection (2016, methodology): “Context information has been modeled by a recurrent neural network in [26] and acquired from multiple regions around the object location in [7,5,27].”
Attend Refine Repeat: Active Box Proposal Generation via In-Out Localization (2016, neutral): “Note that active localization schemes have also been previously applied in the object detection literature [4, 13, 14, 17, 30, 39, 42].”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, background): “, [24, 81, 75], and our model is also able to learn this.”
Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data (2018, background): “Other researchers [25], [26] have also found context to be important.”
Weakly Supervised Object Localization and Detection: A Survey (2021, unknown): “…efforts for simultaneously accomplishing multiple afore-mentioned tasks have been made in the conventional fully supervised learning scenario [57], [78], [174], [213], [220], which have demonstrated that such learning mechanism can bring helpful information from one task to the other ones.”
Joint Object and Part Segmentation Using Deep Learned Potentials (2015, background): “Recently, deep convolutional neural networks [24] (CNN) have achieved great success in many applications such as object detection [17, 22, 47] and end-to-end segmentation [40, 12, 21, 27, 7], with advanced network structures such as the VGG-Net [33].”
Contextual Priming and Feedback for Faster R-CNN (2016, methodology): “, CRFs are commonly used to incorporate context and post-process segmentation outputs [70–72] or to jointly reason about regions, segmentation and detection [66, 73].”
Zoom Better to See Clearer: Human and Object Parsing with Hierarchical Auto-Zoom Net (2015, background): “These include exploiting multiple cues [14], hierarchical region grouping [2,11], and applying general or salient object proposals combined with iterative localization [1,37,32].”
Bayesian Relational Memory for Semantic Visual Navigation (2019, background): “Similar ideas of using semantic knowledge to enhance reasoning have been applied to image classification [34, 56], segmentation [52, 64], situation recognition [33], visual question answering [57, 9, 53, 28, 26], image retrieval [29] and relation detection [62].”
Scale-Aware Fast R-CNN for Pedestrian Detection (2015, background): “Yan et al. [43] proposed to map the pedestrians in different resolutions to a common subspace to reduce the differences of local features.”
Deep learning for visual understanding: A review (2016, methodology): “RCNNs improved the benchmark by a large margin, and became the base model for many other promising algorithms [164,177,178,183,185].”
Zoom Better to See Clearer: Human Part Segmentation with Auto Zoom Net (2015, background): “…multiple cues, such as using 3D scene layouts (Hoiem et al., 2008), hierarchical region grouping (Arbelaez et al., 2011; Florack et al., 1996), and applying general or salient object proposals by using iterative localization, e.g., (Alexe et al., 2012; Zhu et al., 2015; Wang et al., 2012).”
Multi-scale Patch Aggregation (MPA) for Simultaneous Detection and Segmentation (2016, methodology): “We also report results on Microsoft COCO test-std/test-dev dataset in this paper.”