Learning to Segment Every Thing

By Ronghang Hu, Piotr Doll'ar, Kaiming He, Trevor Darrell, Ross B. Girshick. 2017

308 citations recorded by the source corpus.

Most methods for object instance segmentation require all training examples to be labeled with segmentation masks. This requirement makes it expensive to annotate new categories and has restricted instance segmentation models to ~100 well-annotated classes. The goal of this paper is to propose a new partially supervised training paradigm, together with a novel weight transfer function, that enables training instance segmentation models on a large set of categories all of which have box annotations, but only a small fraction of which have mask annotations. These contributions allow us to train Mask R-CNN to detect and segment 3000 visual concepts using box annotations from the Visual Genome dataset and mask annotations from the 80 classes in the COCO dataset. We evaluate our approach in a controlled study on the COCO dataset. This work is a first step towards instance segmentation models that have broad comprehension of the visual world.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Unified Perceptual Parsing for Scene Understanding (2018, background): “Another recent work [3] proposes a partially supervised training paradigm to scale up the segmentation of objects to 3 , 000 objects using box annotations only.”
Auto-Encoding Scene Graphs for Image Captioning (2018, background): “This situation is particularly embarrassing in front of the booming “mid-level” vision techniques nowadays: we can already detect and segment almost everything in an image [12, 17, 36].”
Meta R-CNN: Towards General Solver for Instance-Level Low-Shot Learning (2019, background): “It is worth noting that, as Meta R-CNN solely remodels the predictor branches into a meta-learner, it potentially can be extended to a broad range of models [15, 20, 24, 45] in the entire R-CNN family.”
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation (2021, methodology): “[19] proposed an approach for predicting unknown object masks using known object bounding boxes.”
Image Segmentation Using Deep Learning: A Survey (2020, background): “[65] proposed a new partially-supervised training paradigm together with a novel weight transfer function,”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, methodology): “Our proposed method can also be used in a semi-supervised scenario (with a mixture of fully-and weakly-labelled training examples), but unlike [44], our approach works with only weak supervision as well.”
Learning by Abstraction: The Neural State Machine (2019, methodology): “generation [84, 85, 16, 88]. Here, we largely follow the approaches of Yang et al. [85] and Chen et al. [16] in conjunction with a variant of the Mask R-CNN object detector [34] proposed by Hu et al. [39]. Further details regarding the graph generation can be found in section 7.4. By using such a gr”
BoxInst: High-Performance Instance Segmentation with Box Annotations (2020, methodology): “Following previous works [14, 20, 33] in this setting, we conduct the experiments on the COCO dataset and split the 80 classes in COCO into two groups – 20 classes present in Pascal VOC and 60 classes not in Pascal VOC.”
CANet: Class-Agnostic Segmentation Networks With Iterative Refinement and Attentive Few-Shot Learning (2019, methodology): “, interactive segmentation with click or scribble annotations [36, 14] and instance segmentation with bounding box or extreme point priors [10, 21].”
Hierarchy Parsing for Image Captioning (2019, methodology): “Here we utilize the partially supervised training paradigm [12] to train Mask R-CNN, enabling instance segmentation over the entire 1,600 objects.”
Activation Functions: Comparison of trends in Practice and Research for Deep Learning (2018, background): “…across different domains from object recognition and classiﬁcation [8], [10], [25], [26], to speech recognition [27], [28], segmentation [29], [30], scene understanding and description [31], [32], machine translation [33], [34] test to speech systems [35], [36], cancer detection systems [37],…”
Bayesian Semantic Instance Segmentation in Open Set World (2018, methodology): “, [5,13]) when tested on known object classes, while it is able to segment miss-detected and unknown objects.”
Auto-FPN: Automatic Network Architecture Adaptation for Object Detection Beyond Classification (2019, methodology): “For Visual Genome (VG) [27], we use the synsets [52] instead of the raw names of the categories due to inconsistent label annotations, following [22, 12, 23].”
Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling (2021, background): “Given bounding box annotations, [15, 20, 21, 30, 31] exploit pixelwise similarity to infer object masks while [18, 19, 32, 33] learn to share mask knowledge between mask and box supervision to enhance performances.”
Learning to Segment the Tail (2020, methodology): “, Mask R-CNN [17]) on the Mask R-CNN [13] code base maskrcnn benchmark(3).”
UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation (2019, background): “Skip connections have also played a key role in the success of instance-level segmentation models such as [12], [13] where the idea is to segment and distinguish each instance of desired objects.”
Faithful Multimodal Explanation for Visual Question Answering (2018, methodology): “In order to ground the explanations to a more precise visual regions, we adopt instance segmentation (Hu et al. 2018) to segment out objects in over 3,000 common categories instead of bounding-box detection used in the original BUTD model.”