What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis

By Jeonghun Baek, Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Seong Joon Oh, Hwalsuk Lee. IEEE International Conference on Computer Vision, 2019

570 citations recorded by the source corpus.

Many new proposals for scene text recognition (STR) models have been introduced in recent years. While each claim to have pushed the boundary of the technology, a holistic and fair comparison has been largely missing in the field due to the inconsistent choices of training and evaluation datasets. This paper addresses this difficulty with three major contributions. First, we examine the inconsistencies of training and evaluation datasets, and the performance gap results from inconsistencies. Second, we introduce a unified four-stage STR framework that most existing STR models fit into. Using this framework allows for the extensive evaluation of previously proposed STR modules and the discovery of previously unexplored module combinations. Third, we analyze the module-wise contributions to performance in terms of accuracy, speed, and memory demand, under one consistent set of training and evaluation datasets. Such analyses clean up the hindrance on the current comparisons to understand the performance gain of the existing modules. Our code is publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9387ffc038bc744e5eb335fe54b4f3f184202d77

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CORD: A Consolidated Receipt Dataset for Post-OCR Parsing (2019, background): “Recently, deep learning in computer vision domain has significantly improved the performances of OCR [3, 4].”
RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition (2020, methodology): “Later, top-down methods [12,40,15,11,24,41,3,42,57,8,7,25,50,35,1,55,45,16] were proposed, in which text sequences are end-to-end predicted without the single character detection.”
Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition (2021, unknown): “Existing methods mainly treat STR as a visual recognition (VR) task and perform character-level recognition on input images, including visual-text sequence translationbased methods (Yang et al. 2017; Shi et al. 2018; Baek et al. 2019; Li et al. 2019; Litman et al. 2020) and semantic segmentation-bas”
TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models (2021, methodology): “(Shi et al., 2016b; Baek et al., 2019; Litman et al., 2020; Shi et al., 2018; Zhan and Lu, 2019) address this problem by processing the input images with an initial rectification step.”
LaTr: Layout-Aware Transformer for Scene-Text VQA (2021, background): “1 (b) we visualize the information extracted by the optical character recognition (OCR) system [1, 6, 13, 36], exhibiting three question categories: the ﬁrst type can be answered with just the text tokens; the second type can be answered with text and layout information ( right vs left ); the third…”
OCR-Free Document Understanding Transformer (2021, methodology): “For example, a recent advanced OCR-based model [4,3] requires more than 80M parameters.”
TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text (2021, methodology): “84 9 TPS-ResNet-BiLSTM-Attn [1] ✓ S90k+ST 87.”
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (2022, methodology): “We utilize the CRAFT (Baek et al., 2019) model to recognize the texts in the chart elements.”
Confidence-aware Non-repetitive Multimodal Transformers for TextCaps (2020, unknown): “We use pretrained four-stage STR framework (Baek et al. 2019a) for text recognition.”
Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model (2020, background): “…language processing (NLP) such as classiﬁcation (Zhang et al., 2015), sequence tagging (Guo et al., 2019a), question answering (He and Golub, 2016), and scene text recognition (Baek et al., 2019; Hwang and Sung, 2016), with its simplicity on generating text and its adaptability to other languages.”
Multiple Transformer Mining for VizWiz Image Caption (2021, methodology): “The effectiveness of OCR in image captioning is approved [4], thus several open source OCR solutions have been evaluated, and ﬁnally [2] and [1] are leveraged in our method.”
Brains and algorithms partially converge in natural language processing (2022, methodology): “To model visual representations, every word presented to the subjects was rendered on a gray 100 ×32 pixel background with a centered black Arial font, and input to a VGG network pretrained to recognize words from images 85 , resulting in an 888-dimensional embedding.”