Adversarial Complementary Learning for Weakly Supervised Object Localization

By Xiaolin Zhang, Yunchao Wei, Jiashi Feng, Yi Yang, Thomas S. Huang. 2018

621 citations recorded by the source corpus.

In this work, we propose Adversarial Complementary Learning (ACoL) to automatically localize integral objects of semantic interest with weak supervision. We first mathematically prove that class localization maps can be obtained by directly selecting the class-specific feature maps of the last convolutional layer, which paves a simple way to identify object regions. We then present a simple network architecture including two parallel-classifiers for object localization. Specifically, we leverage one classification branch to dynamically localize some discriminative object regions during the forward pass. Although it is usually responsive to sparse parts of the target objects, this classifier can drive the counterpart classifier to discover new and complementary object regions by erasing its discovered regions from the feature maps. With such an adversarial learning, the two parallel-classifiers are forced to leverage complementary object regions for classification and can finally generate integral object localization together. The merits of ACoL are mainly two-fold: 1) it can be trained in an end-to-end manner; 2) dynamically erasing enables the counterpart classifier to discover complementary object regions more effectively. We demonstrate the superiority of our ACoL approach in a variety of experiments. In particular, the Top-1 localization error rate on the ILSVRC dataset is 45.14%, which is the new state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, methodology): “CutMix also achieves comparable localization accuracies on CUB200-2011 and ImageNet, even when compared against the dedicated state-of-the-art WSOL methods [51, 32, 48, 49, 2] that focus on learning spatially dispersed representations.”
Self-produced Guidance for Weakly-supervised Object Localization (2018, methodology): “…[41] 61.31 50.55 AlexNet-GAP [1] 67.19 52.16 VGGnet-GAP [1] 57.20 45.14 GoogLeNet-GAP [1] 56.40 43.00 GoogLeNet-HaS-32 [2] 54.53 - VGGnet-ACoL [20] 54.17 40.57 GoogLeNet-ACoL [20] 53 baseline methods on the ILSVRC 2016 dataset, as it has more than 1.2 million images of 1,000 classes for training.”
Information-Bottleneck Approach to Salient Region Discovery (2019, background): “Object detection and reconstruction of object shape in the context of this task is frequently achieved based on the knowledge of image labels alone (Hou et al., 2018; Wei et al., 2017; Zhang et al., 2018; Li et al., 2018; Kolesnikov & Lampert, 2016).”
Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation (2021, methodology): “We produce localization maps from the class-specific feature maps of the last convolutional layer, which have been proven by (Zhang et al. 2018) to be mathematically equivalent to CAMs.”
Attention-Based Dropout Layer for Weakly Supervised Object Localization (2019, methodology): “Furthermore, with ADL, the most discriminative region can be identified and erased efficiently, without auxiliary classifiers [17, 59], re-training [49], or additional forward-backward propagation [20].”
Self-Erasing Network for Integral Object Attention (2018, methodology): “We discard the last three fully-connected layers and connect three convolutional layers with 512 channels and kernel size 3 to the backbone as in [39].”
FickleNet: Weakly and Semi-Supervised Semantic Image Segmentation Using Stochastic Inference (2019, methodology): “ACoL [35] and TPL [14] use a classifier to identify the discriminative parts of an object and erase them based on features.”
Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation (2018, background): “Weakly-supervised image recognition approaches [11, 15, 16, 19, 23, 38–42 , 44] Recently, some top-down approaches [43,46] propose to between the state-of-the-art CAM [46] (the 2nd row) and ours (the last row) on quality of the produced object localization maps.”
LayerCAM: Exploring Hierarchical Class Activation Maps for Localization (2021, methodology): “Zhang et al. [8] used two CNN classiﬁcation branches to ﬁnd more conﬁdent areas from class activation maps based on the erasing strategy.”
Crossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language Modeling (2021, background): “The ﬁrst is the “GT-known” [12,50,67,68], where models know the ground-truth class to localize.”
Coarse2Fine: a two-stage training method for fine-grained visual classification (2019, methodology): “To have a fair comparison, we use the localization metric introduced in [30].”
Weakly Supervised Object Localization and Detection: A Survey (2021, neutral): “In [206], Zhang et al. present a CAM-based network architecture which contains a classiﬁcation branch and a counterpart clas-siﬁer branch for object localization.”
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling (2021, background): “The “GT-known” [57,75,76,15] setting aims to localize a given ground-truth class.”
Multi-branch and Multi-scale Attention Learning for Fine-Grained Visual Categorization (2020, methodology): “ig.4) and experimental results prove the effectiveness of these methods to improve object location accuracy. Our weakly supervised object location method can achieve higher location accuracy than ACOL[20] and ADL[21], etc. without adding trainable parameters, we will demonstrate it’s performance in ”
Weakly-Supervised Semantic Segmentation via Sub-Category Exploration (2020, background): “A number of variants [44, 25] have been proposed to extend the initial response via an adversarial erasing strategy in an end-to-end training manner, yet such strategies may gradually expand their attention to non-object regions, leading to inaccurate attention maps.”
SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation (2018, background): “Zhou [28] and Zhang [26, 27] propose to discover precise object regions using a classification network with only image-level labels.”
Simultaneously Localize, Segment and Rank the Camouflaged Objects (2021, background): “[61] erased the high activation area iteratively to force a CNN to learn all relevant features and therefore expanded the discriminative region.”
See Better Before Looking Closer: Weakly Supervised Data Augmentation Network for Fine-Grained Visual Classification (2019, background): “Zhang et al. proposed Adversarial Complementary Learning (ACoL) [36] approach to discover entire objects by training two adversary complementary classiﬁers, which can locate different object’s parts and discover the complementary regions that belong to the same object.”
Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation (2020, methodology): “In line with conventions [23,71,83], our image classifier is based on ImageNet [29] pre-trained VGG-16 [56].”