Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection

By Debidatta Dwibedi, Ishan Misra, Martial Hebert. 2017

709 citations recorded by the source corpus.

A major impediment in rapidly deploying object detection models for instance detection is the lack of large annotated datasets. For example, finding a large labeled dataset containing instances in a particular kitchen is unlikely. Each new environment with new instances requires expensive data collection and annotation. In this paper, we propose a simple approach to generate large annotated instance datasets with minimal effort. Our key insight is that ensuring only patch-level realism provides enough training signal for current object detector models. We automatically ‘cut’ object instances and ‘paste’ them on random backgrounds. A naive way to do this results in pixel artifacts which result in poor performance for trained models. We show how to make detectors ignore these artifacts during training and generate data that gives competitive performance on real data. Our method outperforms existing synthesis approaches and when combined with real images improves relative performance by more than 21% on benchmark datasets. In a cross-domain setting, our synthetic data combined with just 10% real data outperforms models trained on all real data.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “Creating synthetic datasets by inserting objects onto backgrounds has been used for training [75, 16, 21] and evaluating models [36], but previous works do not systematically vary object size, location or orientation, or analyze translation and rotation robustness only at the image level [18].”
Randaugment: Practical automated data augmentation with a reduced search space (2019, background): “Object-centric cropping is commonly used for object detection tasks [24], whereas [8] adds new objects on training images by cut-and-paste.”
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, methodology): “For object detection, object insertion methods [5, 4] have been proposed as a way to synthesize objects in the background.”
Occluded Video Instance Segmentation: A Benchmark (2021, background): “Someworks (DeVries &Taylor, 2017; Yun et al., 2019; Dwibedi et al., 2017; Ghiasi et al., 2021) have proposed augmenting the common datasets (e.”
Learning Data Augmentation Strategies for Object Detection (2019, background): “In addition to cropping and erasing, [7] adds new objects on training images by cut-and-paste.”
InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting (2019, methodology): “[16] improved instance detection by simple cut-and-paste strategy with extra instances that have annotated masks.”
Modeling Visual Context is Key to Augmenting Object Detection Datasets (2018, methodology): “Following [14], we also considered “distractors”, which are then labeled as background.”
BOP Challenge 2020 on 6D Object Localization (2020, methodology): “In contrast, DNN-based methods require large amounts of annotated training images, which have been typically obtained by OpenGL rendering of the 3D object models on top of random backgrounds [31,43,19,11].”
Tradeoffs in Data Augmentation: An Empirical Study (2021, neutral): “, 2016) or cut-and-paste new objects (Dwibedi et al., 2017; Fang et al., 2019; Ngiam et al., 2019).”
Deep Template-based Object Instance Detection (2021, methodology): “Finally, we apply motion blur to the image 20% of the time by convolving a line kernel to the image, as in [8].”
BusterNet: Detecting Copy-Move Image Forgery with Source/Target Localization (2018, methodology): “Inspired by [13] and also [10], we start with an original image X with an associated object mask Ms, randomly generate an affine matrix m to transform both the source mask and the source object image.”
Distilling Localization for Self-Supervised Representation Learning (2020, background): “When copy-and-pasting an object to a background, blending has proven to be important for object detection [14].”
Randomness In Neural Network Training: Characterizing The Impact of Tooling (2021, methodology): “…, 2010, He et al., 2016] Data augmentation stochastic transformations to the input data [Kukaˇcka et al., 2017, Hernández-García and König, 2018, Dwibedi et al., 2017, Zhong et al., 2017] Data shufﬂing inputs shufﬂed randomly and batched during training [Smith et al., 2018] Stochastic Layers…”
RandAugment: Practical data augmentation with no separate search (2019, background): “Object-centric cropping is commonly used for object detection tasks [30], whereas [9] adds new objects on training images by cut-and-paste.”
Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-Modal Pretraining (2021, methodology): “With real-world background images from Places365 [51], a copy-and-paste augmentation [10] is applied to these foreground masks and background images to generate synthesized images.”
Learning to Segment via Cut-and-Paste (2018, background): “, [19], which takes existing masks and creates more training data out of it.”
ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning (2020, background): “ClassMix also shares similarities with other segmentation-based augmentation strategies [9, 32, 8, 35, 11], where annotated single instances of objects are cut out of images, and pasted onto new background scenes.”