DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision

By Shiyi Lan, Zhiding Yu, Christopher Bongsoo Choy, Subhashree Radhakrishnan, Guilin Liu, Yuke Zhu, Larry S. Davis, Anima Anandkumar. 2021

96 citations recorded by the source corpus.

We introduce DiscoBox, a novel framework that jointly learns instance segmentation and semantic correspondence using bounding box supervision. Specifically, we propose a self-ensembling framework where instance segmentation and semantic correspondence are jointly guided by a structured teacher in addition to the bounding box supervision. The teacher is a structured energy model incorporating a pairwise potential and a cross-image potential to model the pairwise pixel relationships both within and across the boxes. Minimizing the teacher energy simultaneously yields refined object masks and dense correspondences between intra-class objects, which are taken as pseudo-labels to supervise the task network and provide positive/negative correspondence pairs for dense contrastive learning. We show a symbiotic relationship where the two tasks mutually benefit from each other. Our best model achieves 37.9% AP on COCO instance segmentation, surpassing prior weakly supervised methods and is competitive to supervised methods. We also obtain state of the art weakly supervised results on PASCAL VOC12 and PF-PASCAL with real-time inference.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Lei Zhang
3. Jianwei Yang
4. Ziwei Liu
5. Chen Change Loy
6. Mohit Bansal
7. Devi Parikh
8. Zhe Gan

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Lei Zhang
3. Ming-Hsuan Yang
4. Dacheng Tao
5. Chen Change Loy
6. Xiaogang Wang
7. Trevor Darrell
8. Jitendra Malik

In Representation & Self-Supervised Learning

1. Jianfeng Gao
2. Ziwei Liu
3. Chen Change Loy
4. Antonio Torralba
5. Sanja Fidler
6. Jianwei Yang
7. Siyi Liu
8. Huan Ling

Citations in the corpus

• A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Prominent methods design teacher models or weak supervision loss, like BoxInst [42], Box2Mask [30], DiscoBox [25] and Mask AutoLabelers [24].”
• Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling (2021, background): “Given bounding box annotations, [15, 20, 21, 30, 31] exploit pixelwise similarity to infer object masks while [18, 19, 32, 33] learn to share mask knowledge between mask and box supervision to enhance performances.”
• Transformer-Based Visual Segmentation: A Survey (2023, methodology): “For weakly supervised instance segmentation, previous works [308], [309], [310] mainly leverage the box priors to supervise mask heads.”
• BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations (2022, background): “Reducing labeling costs can be achieved in a variety of ways, such as interactive humanin-the-loop annotation [2, 50, 53, 64, 73], nearest-neighbor mask transfer [26, 51, 63, 70], or using weak supervision in the form of cheaper labels such as object boxes [7, 43, 45], scribbles [48, 69] or very coa”
• Box-supervised Instance Segmentation with Level Set Evolution (2022, methodology): “Following [25, 26], we also report the average precision (AP) at four IoU thresholds (including 0.”