BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation

By Jifeng Dai, Kaiming He. 2015

1,111 citations recorded by the source corpus.

Recent leading approaches to semantic segmentation rely on deep convolutional networks trained with human-annotated, pixel-level segmentation masks. Such pixel-accurate supervision demands expensive labeling effort and limits the performance of deep networks that usually benefit from more training data. In this paper, we propose a method that achieves competitive accuracy but only requires easily obtained bounding box annotations. The basic idea is to iterate between automatically generating region proposals and training convolutional networks. These two steps gradually recover segmentation masks for improving the networks, and vise versa. Our method, called "BoxSup", produces competitive results (e.g., 62.0% mAP for validation) supervised by boxes only, on par with strong baselines (e.g., 63.8% mAP) fully supervised by masks under the same setting. By leveraging a large amount of bounding boxes, BoxSup further yields state-of-the-art results on PASCAL VOC 2012 and PASCAL-CONTEXT [26].

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Fully convolutional networks for semantic segmentation (2014, background): “ have been extended to new tasks and data. Tasks include region proposals [31], contour detection [32], depth regression [33], optical ﬂow [34], and weaklysupervised semantic segmentation [35], [36], [37], [38]. In addition, new works have improved the FCNs presented here to further advance the stat”
Fully Convolutional Instance-Aware Semantic Segmentation (2016, background): “As the per-pixel category labeling is expensive, the supervision signals in FCNs have been relaxed to boxes [6], scribbles [23], or weakly supervised image class labels [19, 20].”
Instance-Aware Semantic Segmentation via Multi-task Network Cascades (2015, background): “Since the development of fully convolutional networks (FCNs) [23], the accuracy of semantic segmentation has been improved rapidly [5, 24, 6, 31] thanks to deeply learned features [20, 27], large-scale annotations [22], and advanced reasoning over graphical models [5, 31].”
COCO-Stuff: Thing and Stuff Classes in Context (2016, background): “re mutually exclusive and at a comparable level of granularity. Furthermore, while PASCAL Context has a seemingly impressive number of 540 classes, in practice most classes are too rare to be usable: [38,18,37,17,62,5] only use the 60 most frequent classes, out of which only 28 are stuff classes (Ta”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, background): “This direction has been extended by several follow-up papers [17], [40], [58], [59], [60], [61], [62], [63], [65], since the first version of our work was published [38].”
Deep Extreme Cut: From Extreme Points to Object Segmentation (2017, methodology): “Also related to our approach, [7, 17] train semantic segmentation methods from box supervision.”
Conditional Random Fields as Recurrent Neural Networks (2015, methodology): “/a 61.8 FCN-8s [37] n/a 62.7 62.2 Hypercolumn[24] n/a n/a 62.6 Zoomout [38] 64.4 64.1 64.4 Context-DeepCNN-CRF [35] n/a n/a 70.7 DeepLabMSc [10] n/a n/a 71.6 Our method w/o COCO 73.6 72.4 72.0 BoxSup [12] n/a n/a 71.0 DeepLab [10, 41] n/a n/a 72.7 Our method with COCO 75.7 75.0 74.7 Table 2. Mean IU”
Rethinking Atrous Convolution for Semantic Image Segmentation (2017, background): “, networks like [27, 74]) due to limited GPU memory and thus it is usually applied during the inference stage [14].”
Learning Deconvolution Network for Semantic Segmentation (2015, methodology): “Note that only the images in PASCAL VOC 2012 augmented datasets are used for training in our experiment, whereas some state-of-theart algorithms [2, 21] also employ Microsoft COCO to improve performance.”
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation (2015, methodology): “DeepLab-CRF in [3] trains FCNNs and applies a dense CRF [24] method as a post-processing step.”
Semantic Image Segmentation via Deep Parsing Network (2015, methodology): “Following [39, 5], we pre-train DPN with COCO, where 20 object categories that are also presented in VOC12 are selected for training.”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “Certain recent works such as [20, 77, 78] manage to im-pute missing data in an EM-type approach, by exploiting domain-speciﬁc knowledge - e.g. by requesting that a ﬁxed percentage of the pixels contained in the bounding box of an object obtain the same label as the object.”
Attention to Scale: Scale-Aware Semantic Image Segmentation (2015, methodology): “Following the same experimental protocol [11, 15, 58], we augment the original training set from the annotations by [26].”
Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network (2017, methodology): “FCN-8s-heavy [29] 67.2 TTI zoomout v2 [26] 69.6 MSRA BoxSup [9] 71.0 DeepLab-MSc-CRF-LargeFOV [6] 71.6 Oxford TVG CRF RNN COCO [37] 74.7 CUHK DPN COCO [24] 77.5 Oxford TVG HO CRF [2] 77.9 CASIA IVA OASeg [33] 78.3 Adelaide VeryDeep FCN VOC [34] 79.1 LRR 4x ResNet COCO [12] 79.3 Deeplabv2-CRF [7]…”
Incremental Learning in Semantic Segmentation from Image Labels (2021, methodology): “To address this issue, Weakly Supervised Semantic Segmentation (WSSS) methods aim to obtain effective segmentation models using cheaper supervisions such as bounding boxes [18, 32, 49], scribbles [39, 60], points [6, 53], and image-level labels [31, 34, 35, 59].”
Accelerating Very Deep Convolutional Networks for Classification and Detection (2015, neutral): “detection [2], [9], [10], [11], semantic segmentation [12], [13], [14], [40], [41], image captioning [42], [43], [44], video/action recognition [45], image question answering [46], texture recognition [47], etc.”
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation (2016, methodology): “If not further specified, we apply test-time multi-scale evaluation, which is a common practice in segmentation methods [10, 6].”