Fully Convolutional Multi-Class Multiple Instance Learning

By Deepak Pathak, Evan Shelhamer, Jonathan Long, Trevor Darrell. 2014

326 citations recorded by the source corpus.

Multiple instance learning (MIL) can reduce the need for costly annotation in tasks such as semantic segmentation by weakening the required degree of supervision. We propose a novel MIL formulation of multi-class semantic segmentation learning by a fully convolutional network. In this setting, we seek to learn a semantic segmentation model from just weak image-level labels. The model is trained end-to-end to jointly optimize the representation while disambiguating the pixel-image label assignment. Fully convolutional training accepts inputs of any size, does not need object proposal pre-processing, and offers a pixelwise loss map for selecting latent instances. Our multi-class MIL loss exploits the further supervision given by images with multiple labels. We evaluate this approach through preliminary experiments on the PASCAL VOC segmentation challenge.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Webly Supervised Learning of Convolutional Networks (2015, methodology): “The outline of our approach is shown in Figure 2.”
The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, neutral): “Last but not least, several studies [5,11,48–50,53,74,76] lately have explored different forms of weak supervision, such as bounding boxes or image-level labels, for training CNNs for pixel-level semantic labeling.”
Deep Extreme Cut: From Extreme Points to Object Segmentation (2017, methodology): “Image-level labels [26], noisy web labels [1, 16] and scribble-level labels [20] are some of the supervisory signal that have been used to guide segmentation methods.”
Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation (2015, methodology): “MIL It is instructive to compare our EM-based approach with two recent Multiple Instance Learning (MIL) methods for learning semantic image segmentation models [30, 31].”
Constrained Convolutional Neural Networks for Weakly Supervised Segmentation (2015, methodology): “Recently, there has been a series of work in tackling this problem using convolutional neural networks [23, 25, 26] which we discussed in Section 1.”
Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation (2015, methodology): “In weakly-supervised setting, the models for semantic segmentation have been trained with only image-level labels [7, 8, 9] or bounding box class labels [10].”
ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation (2016, methodology): “There have been a series of methods [25, 24, 7, 27] on weakly-supervised learning CNN/FCN for semantic segmentation.”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, background): “Learning from weak supervision: Given the known challenges in gathering exhaustive annotations for various image and video tasks, leveraging object labels and other meta information to supply weak supervision [7, 15, 19, 22, 30, 31, 39, 40, 50, 52, 53, 54, 55, 56, 59] is a widely-adopted approach.”
FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation (2016, methodology): “In order to do this, we draw upon recent weak learning literature [25, 26], which introduced a fully convolutional constrained multiple instance learning objective.”
No More Discrimination: Cross City Adaptation of Road Scene Segmenters (2017, background): “Moreover, [24, 26, 27] only require image-level labels during data collection and training.”
A Multi-scale Multiple Instance Video Description Network (2015, background): “In [18], the authors introduce the concept of multiple instance learning with FCN to make use of multi-class image labels for training, but their ultimate goal is still image segmentation.”
Single-Stage Semantic Segmentation From Image Labels (2020, methodology): “training one network [17, 38, 40, 41], they were soon superseded by more advanced pipelines, employing multiple models, training cycles, and off-the-shelf saliency methods [24, 55, 57, 60].”
Weakly Supervised Semantic Segmentation Using Web-Crawled Videos (2017, methodology): “Among many possible choices, image-level labels are of the form requiring the minimum annotation cost thus have been widely used [17, 28, 29, 30].”
Effective Use of Synthetic Data for Urban Scene Semantic Segmentation (2018, background): “Weakly-supervised semantic segmentation aims to leverage a weaker form of annotation, such as image tags [23, 30, 32, 34, 40, 41, 49, 48, 29], bounding boxes [8, 21], scribbles [2] and object size statistics [31], which are cheaper to obtain.”
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos (2018, methodology): “MIL has been a effective framework for weakly-supervised learning in several applications, including image classification [50], object localization [13], tracking [5], and instance segmentation [37].”
Deep Dual Learning for Semantic Image Segmentation (2017, methodology): “Pixel Tag Bbox Scribble MIL-FCN [23] 10k,VA MIL-sppxl [24] 760k, I CCNN [22] 10k,VA WSSL (semi) [21] 15k,VA+C 118k, C BoxSup [4] 10k,VA 133k, C ScribbleSup [13] 11k,VA 10k,V7 DIS (ours) 2.”
Exploiting Saliency for Object Segmentation from Image Level Labels (2017, methodology): “Im ag e la b el s o n ly MIL-FCN [32] I+P 25.”