COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images

By Andreas Veit, Tomas Matera, Lukás Neumann, Jiri Matas, Serge J. Belongie. 2016

623 citations recorded by the source corpus.

This paper describes the COCO-Text dataset. In recent years large-scale datasets like SUN and Imagenet drove the advancement of scene understanding and object recognition. The goal of COCO-Text is to advance state-of-the-art in text detection and recognition in natural images. The dataset is based on the MS COCO dataset, which contains images of complex everyday scenes. The images were not collected with text in mind and thus contain a broad variety of text instances. To reflect the diversity of text in natural scenes, we annotate text with (a) location in terms of a bounding box, (b) fine-grained classification into machine printed text and handwritten text, (c) classification into legible and illegible text, (d) script of the text and (e) transcriptions of legible text. The dataset contains over 173k text annotations in over 63k images. We provide a statistical analysis of the accuracy of our annotations. In addition, we present an analysis of three leading state-of-the-art photo Optical Character Recognition (OCR) approaches on our dataset. While scene text detection and recognition enjoys strong advances in recent years, we identify significant shortcomings motivating future work.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “In general image-caption datasets [45, 29, 54, 41, 43], many image-text pairs contain either no scene text-related visual regions or no scene text-related language referring, and are thus less helpful to Text-VQA/TextCaption.”
All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting (2019, methodology): “Following (Lyu et al. 2018b), our model is not trained with the training set of COCO-Text.”
Towards VQA Models That Can Read (2019, background): “Several existing datasets study text detection and/or parsing in natural everyday scenes: COCO-Text [43], Street-View text [44] IIIT-5k [33] and ICDAR 2015 [22].”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “Examples of this include vision challenges, such as ImageNet (Deng et al., 2009) and COCO (Veit et al., 2016), and game challenges, where simulators produce hundreds of years of experience for online RL agents such as AlphaGo (Silver et al., 2016) and the OpenAI Five (Berner et al., 2019).”
Scene Text Visual Question Answering (2019, methodology): “Text is present in about 50% of the images in large-scale datasets such as MS Common Objects in Context [53] and the percentage goes up sharply in urban environments.”
DeepSeek-VL: Towards Real-World Vision-Language Understanding (2024, neutral): “ArT (Chng et al., 2019) 1.2% MLT-17 (Nayef et al., 2017) LSVT (Sun et al., 2019) UberText (Zhang et al., 2017) Coco-text (Veit et al., 2016) RCTW-17 (Shi et al., 2017) ReCTS (Zhang et al., 2019) TextOCR (Singh et al., 2021) OpenVINO (Krylov et al., 2021) HierText (Long et al., 2022) Document OCR…”
PixelLink: Detecting Scene Text via Instance Segmentation (2018, background): “To further check the generalization ability of PixelLink, the same 2s model is also tested on COCO-Text evaluation set without any ﬁne-tuning, achieving a performance of 35.4, 54.0, 42.4 for recall, precision and F-score, exceeding the corresponding results 32.4, 50.4, 39.5 of EAST (Zhou et al. 2017”
Rosetta: Large Scale System for Text Detection and Recognition in Images (2018, methodology): “initial choice of metric for Faster-RCNN model training was F1-score (a harmonic mean of precision and recall), which was at the initial moment of development a default evaluation metric in COCO-Text [28]. However, we observed that our reported validation set F1-score would fluctuate with a standard”
TextBoxes++: A Single-Shot Oriented Scene Text Detector (2018, methodology): “Methods recall precision f-measure Baseline A [49] 0.”
Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes (2018, methodology): “We also evaluate our method on COCO-Text [18], [70] to”
TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text (2021, methodology): “Row #1-2 show the cross-dataset comparison between COCOText [57] and TextOCR.”
Rotation-Sensitive Regression for Oriented Scene Text Detection (2018, methodology): “COCO-Text dataset [45] is evaluated to verify the accuracy of regression, which provides a evaluation protocol where the IOU threshold is set to 0.”
OCRBench: on the hidden mystery of OCR in large multimodal models (2023, neutral): “…(1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly occluded scene text (WOST) and heavily…”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, neutral): “For scene images, we include COCO-2017 [66], Object365 [56], SA-1B [57], ShareGPT4o [53], ShareGPT4V [63], DenseFusion [64], and LLaVA-Recap (LCS-558K) [28].”
RL Unplugged: A Suite of Benchmarks for Ofﬂine Reinforcement Learning (2020, background): “Examples of this include vision challenges, such as ImageNet [Deng et al., 2009] and COCO [Veit et al., 2016], and game challenges, where simulators produce hundreds of years of experience for online RL agents such as AlphaGo [Silver et al., 2016] and the OpenAI Five [Berner et al., 2019].”
Deep TextSpotter: An End-to-End Trainable Scene Text Localization and Recognition Framework (2017, methodology): “2Method A [27] was authored by Google and neither the training data nor the algorithm is published.”
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning (2022, background): “To build M ULTI I NSTRUCT , we first collect 34 tasks from the existing studies in visual and multimodal learning, covering Visual Question Answering (Goyal et al., 2017; Krishna et al., 2017; Zhu et al., 2016; Hudson and Manning, 2019; Singh et al., 2019; Marino et al., 2019), Commonsense Reasoning”
ABCNet: Real-Time Scene Text Spotting With Adaptive Bezier-Curve Network (2020, methodology): “Datasets: AddF2k [46]; IC13 [16]; IC15 [15]; TT [4]; MLT [31]; COCO-Text [39].”