Synthetic Data for Text Localisation in Natural Images

By Ankush Gupta, Andrea Vedaldi, Andrew Zisserman. 2016

1,563 citations recorded by the source corpus.

In this paper we introduce a new method for text detection in natural images. The method comprises two contributions: First, a fast and scalable engine to generate synthetic images of text in clutter. This engine overlays synthetic text to existing background images in a natural way, accounting for the local 3D scene geometry. Second, we use the synthetic images to train a Fully-Convolutional Regression Network (FCRN) which efficiently performs text detection and bounding-box regression at all locations and multiple scales in an image. We discuss the relation of FCRN to the recently-introduced YOLO detector, as well as other end-to-end object detection systems based on deep learning. The resulting detection network significantly out performs current methods for text detection in natural images, achieving an F-measure of 84.2% on the standard ICDAR 2013 benchmark. Furthermore, it can process 15 images per second on a GPU.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting (2019, background): “SynthText (Gupta, Vedaldi, and Zisserman 2016) has about 800,000 images, which are generated via synthesizing engine.”
Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “various prediction tasks, including gaze estimation [43], text detection and classification in RGB images [9, 15], font recognition [42], object detection [10, 27], hand pose estimation in depth images [38, 37], scene recognition in RGB-D [11], semantic segmentation of urban scenes [31], and human p”
Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model (2021, positive): “Part of it by using the similar methods as TAP, part of it by generating synthetic text images as described in [1].”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “, 2014; 2016) and SynthText (ST) (Gupta et al., 2016) datasets used for training scene text recognition models.”
AutoML: A Survey of the State-of-the-Art (2019, background): “Moreover, natural scene text can also be generated in [51], [52].”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “Synthetic Data Generative Adversarial Networks [57], [58], [59], [60], [61], [62] Policies [63], [64] Image [65], [66], [67], [68], [69], [70], [71] Text [72], [73], [74]”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, background): “To address this, an emerging theme of work [16] moves away from graphics based renderings to composing real images.”
Modeling Visual Context is Key to Augmenting Object Detection Datasets (2018, methodology): “A more ambitious data augmentation technique consists of leveraging segmentation annotations, either obtained manually, or from an automatic segmentation system, and create new images with objects placed at various positions in existing scenes [14,15,16].”
Learning 3D Object Categories by Looking Around Them (2017, methodology): “Inspired by the approach of [12], we propose to solve”
Sliding Line Point Regression for Shape Robust Scene Text Detection (2018, methodology): “Some of the previous methods [1] [2] have obtained good results on many horizontal scene texts dataset based on Faster R-CNN [3] or SSD [4].”
Efficient and Accurate Arbitrary-Shaped Text Detection With Pixel Aggregation Network (2019, background): “SynthText [10] is a large scale synthetically generated dataset containing 800K synthetic images.”
Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training (2017, background): “Despite the widespread use of synthetic data for training deep networks for real world images [29, 8, 31, 20], its use for medical imaging applications has been relatively limited.”
EAST: An Efficient and Accurate Scene Text Detector (2017, methodology): “features are manually designed [5, 25, 40, 10, 26, 45] to capture the properties of scene text, while in deep learning based methods [3, 13, 11, 12, 7, 48] effective features are directly learned from training data.”
Synthetic Data for Deep Learning (2019, methodology): “hine-printed forms, with Otsu binarization applied before pasting. There exist published datasets of synthetic text and software to produce them, in particular MJSynth [276] and SynthText in the Wild [222]. In the latter work, Gupta et al. use available depth estimation and segmentation solutions to”
Look More Than Once: An Accurate Detector for Text of Arbitrary Shapes (2019, methodology): “At warming-up step, we train DR branch using synthetic dataset [4] only for 10 epochs.”
Cascaded Segmentation-Detection Networks for Text-Based Traffic Sign Detection (2018, methodology): “First, the model is trained using 800k synthetic images from [57] for 50k iterations.”