Exploit Bounding Box Annotations for Multi-Label Object Recognition

By Hao Yang, Joey Tianyi Zhou, Yu Zhang, Bin-Bin Gao, Jianxin Wu, Jianfei Cai. 2015

181 citations recorded by the source corpus.

Convolutional neural networks (CNNs) have shown great performance as general feature representations for object recognition applications. However, for multi-label images that contain multiple objects from different categories, scales and locations, global CNN features are not optimal. In this paper, we incorporate local information to enhance the feature discriminative power. In particular, we first extract object proposals from each image. With each image treated as a bag and object proposals extracted from it treated as instances, we transform the multi-label recognition problem into a multi-class multi-instance learning problem. Then, in addition to extracting the typical CNN feature representation from each proposal, we propose to make use of ground-truth bounding box annotations (strong labels) to add another level of local information by using nearest-neighbor relationships of local regions to form a multi-view pipeline. The proposed multi-view multiinstance framework utilizes both weak and strong labels effectively, and more importantly it has the generalization ability to even boost the performance of unseen categories by partial strong labels from other categories. Our framework is extensively compared with state-of-the-art handcrafted feature based methods and CNN based methods on two multi-label benchmark datasets. The experimental results validate the discriminative power and the generalization ability of the proposed framework. With strong labels, our framework is able to achieve state-of-the-art results in both datasets.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, methodology): “On the one hand, some papers report performance in terms of mean average precision [50, 51, 55] (reminiscent of object detection) while, on the other hand, others consider intersection-over-union based metrics such as F1 score [12, 24, 57, 7, 6, 49, 48, 61, 38, 39].”
Multi-label Image Recognition by Recurrently Discovering Attentional Regions (2017, methodology): “We employ the Euclidean loss as the objective function following [27, 30].”
Action Recognition in Still Images With Minimum Annotation Efforts (2016, background): “, riding, phoning), existing works usually also require manually annotated human bounding-boxes as input [7]–[13] during both training and testing stages, which have played a critical role in modelling the typical human poses of different actions, and in characterizing the underlying human-object in”
Detecting 11K Classes: Large Scale Object Detection Without Fine-Grained Bounding Boxes (2019, methodology): “A classical way to aggregate proposal scores is to use max or average pooling [38, 39].”
Attentive Recurrent Neural Network for Weak-supervised Multi-label Image Classification (2018, methodology): “Compared with some region proposal based methods [7, 25, 43] that require the ground truth of bounding boxes which need expensive labor cost and might not be available in practice, our RHN can work well with weak image-level supervisions.”
Order-Free RNN with Visual Attention for Multi-Label Classification (2017, methodology): “Approaches like [31] further utilize additional information like bounding box annotations for learning their models.”
Learning Spatial Regularization with Image-Level Supervisions for Multi-label Image Classification (2017, background): “[42] treated images as a bag of instances/proposals, and solved a multiinstance learning problem.”
Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition (2019, methodology): “To fairly compare with existing methods, we follow them to adopt the average precision (AP) on each category and mean average precision (mAP) over all categories for evaluation [28, 30].”
Multi-Label Image Recognition With Graph Convolutional Networks (2019, unknown): “9 FeV+LV [33] 97.”
Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection (2018, methodology): “These methods always require pre-defined label relations. Some other works ensemble multiple deep models with different input scales [5, 33] while suffering high complexity. Recently, various methods [4, 5, 33, 39, 40, 43, 45] have been proposed to locate semantic regions for learning deep attention”
Asymmetric Loss For Multi-Label Classification (2020, background): “9 FeV+LV [35] 92.”
General Multi-label Image Classification with Transformers (2020, background): “Connecting to Graph Based Neural Relational Learning Another line of recent works employ object localization techniques[56, 52] or attention mechanism[51, 58] or merely get regions of interest roughly due to the lack of label supervision.”
Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition (2017, methodology): “An alternative way (Yang et al. 2016; Wei et al. 2016) is to introduce object proposals that are assumed to contain all possible foreground objects in the image, and aggregate features extracted from all these proposals to incorporate local information for multi-label image recognition.”