From image-level to pixel-level labeling with Convolutional Networks

By Pedro H. O. Pinheiro, Ronan Collobert. 2014

766 citations recorded by the source corpus.

We are interested in inferring object segmentation by leveraging only object class information, and by considering only minimal priors on the object segmentation task. This problem could be viewed as a kind of weakly supervised segmentation task, and naturally fits the Multiple Instance Learning (MIL) framework: every training image is known to have (or not) at least one pixel corresponding to the image class label, and the segmentation task can be rewritten as inferring the pixels belonging to the class of the object (given one image, and its object class). We propose a Convolutional Neural Network-based model, which is constrained during training to put more weight on pixels which are important for classifying the image. We show that at test time, the model has learned to discriminate the right pixels well enough, such that it performs very well on an existing segmentation benchmark, by adding only few smoothing priors. Our system is trained using a subset of the Imagenet dataset and the segmentation experiments are performed on the challenging Pascal VOC dataset (with no fine-tuning of the model on Pascal VOC). Our model beats the state of the art results in weakly supervised object segmentation task by a large margin. We also compare the performance of our model with state of the art fully-supervised segmentation approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Segment Object Candidates (2015, background): “vsky et al. [18] demonstrated that they performverywellontheImageNetclassicationbenchmark. Moreover,thesemodelslearnsufciently general image features, which can be transferred to many different tasks [10, 11, 3, 22, 23]. Given an input image patch, our algorithm generates a class-agnostic mask and a”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization (2016, methodology): “Others have investigated similar methods using global max pooling [34] and log-sum-exp pooling [35].”
Learning Deep Features for Discriminative Localization (2015, methodology): “In this work, we show that, using the right architecture, we can generalize this ability beyond just localizing objects, to start identifying exactly which regions of an image are being used for 1Our models are available at: http://cnnlocalization.csail.mit.edu ar X iv :1 51 2.”
Visual Feature Attribution Using Wasserstein GANs (2017, methodology): “Pinpointing all evidence of a class is important for a variety of tasks such as weakly supervised localisation or segmentation of structures [41, 43, 65], and better understanding disease effects, and physiological or pathological processes in medical images [67, 18, 12, 19, 13, 27, 54, 30, 31, 63].”
Incremental Learning in Semantic Segmentation from Image Labels (2021, background): “To reduce the annotation cost, different types of weak supervision have been proposed: bounding boxes [18, 32], scribbles [39, 62], points [16], and image-level labels [34, 50, 52].”
Top-Down Neural Attention by Excitation Backprop (2016, methodology): “We use the evaluation code from Plummer et al. (2015). To generate scored bounding boxes for a NP, we first compute theword attentionmap for eachword in theNPusing our tag classifier.”
Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation (2015, methodology): “MIL It is instructive to compare our EM-based approach with two recent Multiple Instance Learning (MIL) methods for learning semantic image segmentation models [30, 31].”
What's the Point: Semantic Segmentation with Point Supervision (2015, methodology): “elongs to an object. Such priors have been used in segmentation literature for selecting image regions to segment [31], as unary potentials in a conditional random eld model [20], or during inference [25]. However, to the best of our knowledge, we are the rst to employ this directly in the loss to g”
Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks (2017, methodology): “Similar methods were examined with different pooling layers such as global max pooling [10] and log-sum-exp pooling [11].”
Attention-based Deep Multiple Instance Learning (2018, neutral): “MIL pooling Typically, MIL approaches utilize either the mean pooling or the max pooling, while the latter is mostly used (Feng & Zhou, 2017; Pinheiro & Collobert, 2015; Zhu et al., 2017).”
FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation (2016, methodology): “[27] modeled this problem as multiple instance learning (MIL) and reinforce conﬁdent predictions during the learning process.”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, background): “Subsequently, many approaches have achieved high accuracy using only image-level labels [9, 10, 40, 41], bounding boxes [42, 11, 12], scribbles [20] and points [13].”
No More Discrimination: Cross City Adaptation of Road Scene Segmenters (2017, background): “Moreover, [24, 26, 27] only require image-level labels during data collection and training.”
This Looks Like That: Deep Learning for Interpretable Image Recognition (2018, methodology): “Attention mechanisms that identify the most relevant parts of an input for various tasks have been integrated into neural networks: various methods have been proposed to jointly train networks with integrated class-specific attention maps [24, 43].”