InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting

By Haoshu Fang, Jianhua Sun, Runzhong Wang, Minghao Gou, Yong-Lu Li, Cewu Lu. 2019

239 citations recorded by the source corpus.

Instance segmentation requires a large number of training samples to achieve satisfactory performance and benefits from proper data augmentation. To enlarge the training set and increase the diversity, previous methods have investigated using data annotation from other domain (e.g. bbox, point) in a weakly supervised mechanism. In this paper, we present a simple, efficient and effective method to augment the training set using the existing instance mask annotations. Exploiting the pixel redundancy of the background, we are able to improve the performance of Mask R-CNN for 1.7 mAP on COCO dataset and 3.3 mAP on Pascal VOC dataset by simply introducing random jittering to objects. Furthermore, we propose a location probability map based approach to explore the feasible locations that objects can be placed based on local appearance similarity. With the guidance of such map, we boost the performance of R101-Mask R-CNN on instance segmentation from 35.7 mAP to 37.9 mAP without modifying the backbone or network structure. Our method is simple to implement and does not increase the computational complexity. It can be integrated into the training pipeline of any instance segmentation model without affecting the training and inference efficiency. Our code and models have been released at https://github.com/GothicAi/InstaBoost.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “Data augmentation is a widely used method for generating additional data to improve machine learning systems, for image classification [43, 23, 7, 54], object detection [13], instance segmentation [10], and speech recognition [21, 16, 36].”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “For system-level comparison, we adopt an improved HTC [9] (denoted as HTC++) with instaboost [20], stronger multi-scale training [7], 6x schedule (72 epochs), soft-NMS [5], and ImageNet-22K pre-trained model as initialization.”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “Copying-and-pasting foreground instances at different locations into different background scenes has shown to be a simple and effective data augmentation technique for object detection and instance segmentation, as shown in [9,13,82].”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, methodology): “In data pre-processing, Instaboost [23], a multi-scale training [26] with an input image size of 1536 × 1536, a window size of 32 × 32, and a random scale between [0 .”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “For fair comparison, we also use soft-NMS [5], instaboost [25] and a multi-scale training strategy with shorter side in range [400, 1400] while the longer side is no more than 1600.”
Bidirectional Copy-Paste for Semi-Supervised Medical Image Segmentation (2023, unknown): “, instance segmentation [7, 9], semantic segmentation [6, 25] and object detection [5].”
RandAugment: Practical data augmentation with no separate search (2019, methodology): “Data augmentation is a widely used method for generating additional data to improve machine learning systems, for image classification [42, 22, 7, 53], object detection [13], instance segmentation [10], and speech recognition [20, 15, 35].”
Tradeoffs in Data Augmentation: An Empirical Study (2021, unknown): “, 2016) or cut-and-paste new objects (Dwibedi et al., 2017; Fang et al., 2019; Ngiam et al., 2019).”
Vision Transformer Adapter for Dense Predictions (2022, background): “Instaboost [23] augmentation, AdamW [55] optimizer (batch size of 16, initial learning rate of 1×10−4, and weight decay of 0.”
Explicit Shape Encoding for Real-Time Instance Segmentation (2019, background): “Besides, the works on augmenting the performance of instance segmentation frameworks through data augmentation [7, 36], scale normalization [33] can be easily integrated to our system.”
RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, methodology): “These include multi-stage approaches such as Mask R-CNN [20], Mask Scoring (MS) R-CNN [22], HTC [23], Instaboost [87], PointRend [88], SCNet [90], CATNet [5], and HQISNet [1], as well as single-stage methods like SOLO [28], SOLOv2 [89], CondInst [27], BoxInst [91], and Mask2Former [29].”
Learning Object Placement by Inpainting for Compositional Data Augmentation (2020, methodology): “In contrast with this work [7], we can sample directly from the joint distribution of three disjoint variables: object appearance, scene appearance, and stochastic variations of object-scene interaction, without being forced in the original background context.”
ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning (2020, background): “ClassMix also shares similarities with other segmentation-based augmentation strategies [9, 32, 8, 35, 11], where annotated single instances of objects are cut out of images, and pasted onto new background scenes.”
Seesaw Loss for Long-Tailed Instance Segmentation (2020, methodology): “And then, we randomly crop the image to the size of 1280 × 1280 after adopting instaboost augmentation [8].”
An Empirical Study of Detection-Based Video Instance Segmentation (2019, methodology): “In addition to conventional data augmentation strategies ( e.g ., random scaling and cropping), we adopt InstaBoost [11] to further improve data efﬁciency.”
SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation (2020, methodology): “The top-down approaches are divided into two-stage [21,31,8,23,17] and single-stage [13,47,2,42,36] methods.”
k-means Mask Transformer (2022, methodology): “For regularization and augmentations, we adopt drop path [45], random color jittering [29], and panoptic copy-paste augmentation, which is an extension from instance copy-paste augmentation [34,38] by augmenting both ‘thing’ and ‘stuﬀ’ classes.”
Improved YOLOv5 network for real-time multi-scale traffic sign detection (2021, methodology): “Furthermore, the method of annotated instance masks with a location probability map is utilized to augment the training dataset [34], which can effectively enhance the generalization of the dataset.”
Circumventing Outliers of AutoAugment with Knowledge Distillation (2020, background): “Data augmentation has been verified successful in a wide range of visual recognition tasks, including image classification [7,54,52], object detection [35], semantic segmentation [8], person re-identification [57], etc.”