Detect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts

By Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, Alan L. Yuille. 2014

748 citations recorded by the source corpus.

Detecting objects becomes difficult when we need to deal with large shape deformation, occlusion and low resolution. We propose a novel approach to i) handle large deformations and partial occlusions in animals (as examples of highly deformable objects), ii) describe them in terms of body parts, and iii) detect them when their body parts are hard to detect (e.g., animals depicted at low resolution). We represent the holistic object and body parts separately and use a fully connected model to arrange templates for the holistic object and body parts. Our model automatically decouples the holistic object or body parts from the model when they are hard to detect. This enables us to represent a large number of holistic object and body part combinations to better deal with different "detectability" patterns caused by deformations, occlusion and/or low resolution. We apply our method to the six animal categories in the PASCAL VOC dataset and show that our method significantly improves state-of-the-art (by 4.1% AP) and provides a richer representation for objects. During training we use annotations for body parts (e.g., head, torso, etc.), making use of a new dataset of fully annotated object parts for PASCAL VOC 2010, which provides a mask for each part.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

ALICE: Active Learning with Contrastive Natural Language Explanations (2020, neutral): “The off-theshelf semantic segment localization models could be the pre-trained localization models on various large-scale datasets like Visual Genome (Krishna et al., 2017) and PASCAL-Part (Chen et al., 2014).”
Unified Perceptual Parsing for Scene Understanding (2018, background): “Broden uniﬁes several densely labeled image datasets, namely ADE20K [2], Pascal-Context [27], Pascal-Part [28], OpenSurfaces [6], and the Describable Textures Dataset (DTD) [4].”
Semantic Understanding of Scenes Through the ADE20K Dataset (2016, background): “asets with objects, parts and attributes. Two datasets were released that go beyond the typical labeling setup by also providing pixel-level annotation for the object parts, i.e., Pascal-Part dataset [6], or material classes, i.e., OpenSurfaces [2, 3]. We advance this effort by collecting very high-”
Hypercolumns for object segmentation and fine-grained localization (2014, methodology): “We use the part annotations provided by [10].”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “We further perform experiments on semantic part segmentation [98], [99], using the extra PASCAL VOC 2010 annotations by [36].”
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs (2014, methodology): “There are multiple aspects in our model that we intend to refine, such as fully integrating its two main components (CNN and CRF) and train the whole system in an end-to-end fashion, similar to Krähenbühl & Koltun (2013); Chen et al. (2014); Zheng et al. (2015).”
Scene Parsing through ADE20K Dataset (2017, methodology): “Such annotation is only available for the Pascal-Context/Part dataset [21, 5] which contains 40 distinct part classes across 20 object classes.”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, methodology): “mean IoU Deeplab LargeFOV [106] 51.78 Deeplab LargeFOV-CRF [106] 52.95 Multi-scale averaging [15] 54.91 Attention [15] 55.17 Auto Zoom [106] 57.54 Graph-LSTM [60] 60.16 Ours, 1-Task 51.98 Ours, 7-Task 48.82 Table 7: Part segmentation - mean Intersction-Over-Union accuracy on the dataset of [17].”
Attention to Scale: Scale-Aware Semantic Image Segmentation (2015, methodology): “In this section, after presenting the common setting for all the experiments, we evaluate our method on three datasets, including PASCAL-Person-Part [13], PASCAL VOC 2012 [18], and a subset of MS-COCO 2014 [35].”
Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, methodology): “Results based on the Pascal-Part dataset [4] Single-category Multi-category bird cat cow dog horse sheep Avg.”
Interpreting CNNs via Decision Trees (2018, methodology): “Just like in most part-localization studies [7, 38], we used animal categories, which prevalently contain non-rigid shape deformation, for evaluation.”
Interpretable Convolutional Neural Networks (2017, methodology): “Thus, we followed [3, 36] to select the 37 animal categories in the three datasets for evaluation.”
Logic Tensor Networks for Semantic Image Interpretation (2017, methodology): “We artificially add an increasing amount of noise to the PASCAL-PART-dataset training data, and we measure the degradation of the performance, in presence or in absence of axioms.”
Network Dissection: Quantifying Interpretability of Deep Visual Representations (2017, background): “The Broadly and Densely Labeled Dataset (Broden) unifies several densely labeled image data sets: ADE [43], OpenSurfaces [4], Pascal-Context [19], Pascal-Part [6], and the Describable Textures Dataset [7].”
Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning (2016, methodology): “semanticizing CNN patterns: Part localization/detection is an important task in computer vision (Azizpour and Laptev 2012; Simon, Rodner, and Denzler 2014; Li et al. 2013; Chen et al. 2014).”
Interpretability Beyond Classification Output: Semantic Bottleneck Networks (2019, methodology): “Based on the 377 part and material concepts available (351 parts sourced from ADE [32] and Pascal-Part [7] and 26 materials sourced from OpenSurfaces [5]), we compile multiple subsets containing at least 6 concepts in order to explore the relationship between number of concepts and segmentation perf”
Interpreting Deep Visual Representations via Network Dissection (2017, background): “scene 468 ADE [32] 38 object 584 ADE [32], Pascal-Context [34] 491 part 234 ADE [32], Pascal-Part [35] 854 material 32 OpenSurfaces [33] 1,703 texture 47 DTD [36] 140 color 11 Generated 59,250 Fig.”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, background): “Labels for human parts segmentation are acquired from [5], while for saliency and surface normals from [39].”
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, methodology): “We demonstrate the effectiveness of our proposed method on three dense prediction tasks that are well studied in the literature: scene parsing (Cityscapes [18]), person part segmentation (PASCAL-Person-Part [16]), and semantic image segmentation (PASCAL VOC 2012 [24]).”