Deep Structured Output Learning for Unconstrained Text Recognition

By Max Jaderberg, Karen Simonyan, Andrea Vedaldi, Andrew Zisserman. 2014

238 citations recorded by the source corpus.

We develop a representation suitable for the unconstrained recognition of words in natural images: the general case of no fixed lexicon and unknown length. To this end we propose a convolutional neural network (CNN) based architecture which incorporates a Conditional Random Field (CRF) graphical model, taking the whole word image as a single input. The unaries of the CRF are provided by a CNN that predicts characters at each position of the output, while higher order terms are provided by another CNN that detects the presence of N-grams. We show that this entire model (CRF, character predictor, N-gram predictor) can be jointly optimised by back-propagating the structured output loss, essentially requiring the system to perform multi-task learning, and training uses purely synthetically generated data. The resulting model is a more accurate system on standard real-world text recognition benchmarks than character prediction alone, setting a benchmark for systems that have not been trained on a particular lexicon. In addition, our model achieves state-of-the-art accuracy in lexicon-constrained scenarios, without being specifically modelled for constrained recognition. To test the generalisation of our model, we also perform experiments with random alpha-numeric strings to evaluate the method when no visual language model is applicable.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Structured Attention Networks (2017, background): “…applications, most notably in seminal work on graph transformers (LeCun et al., 1998), key work on NLP (Collobert et al., 2011), and in many other areas (Peng et al., 2009; Do & Arti´eres, 2010; Jaderberg et al., 2014; Chen et al., 2015; Durrett & Klein, 2015; Lample et al., 2016, inter alia ).”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “On IIIT5K, which is natural images of individually cropped words, zero-shot CLIP performs a bit more respectively and its performance is similar to Jaderberg et al. (2014) early work combining deep learning and structured prediction to perform openvocabulary OCR.”
Conditional Random Fields as Recurrent Neural Networks (2015, methodology): “ating elements of the CRF model. In [6], the authors combinedaCNNwithHiddenMarkovModelsforthatpurpose, whereasmorerecently,Peng etal .[45]usedamodiedversion of CRFs. Related to this line of works, in [25] a joint CNN and CRF model was used for text recognition on natural images. Tompson et al . [57]”
Recent advances in convolutional neural networks (2015, methodology): “without using lexicon and dictionary, [160] introduces a novel Conditional Random Fields (CRF)-like CNN model to jointly learn character sequence prediction and bigram generation for scene text recognition.”
Human Pose Estimation with Iterative Error Feedback (2015, background): “see references in [26]) but it is a relatively modern topic in conjunction with feature learning, for computer vision [4, 18, 37, 23].”
Structured Prediction Energy Networks (2015, methodology): “It is natural to parametrize the potentials of a CRF using deep features (LeCun et al., 2006; Collobert et al., 2011; Jaderberg et al., 2014; Huang et al., 2015; Chen et al., 2014; Schwing & Urtasun, 2015; Chen et al., 2015).”
Flowing ConvNets for Human Pose Estimation in Videos (2015, background): “Other works have also investigated the beneﬁts of including graphical models in ConvNets [5, 17, 23, 24, 43].”
ESIR: End-To-End Scene Text Recognition via Iterative Image Rectification (2018, background): “For example, [4] adopts a fully connected network to recognize characters, [44] uses CNNs for feature extraction, and [16] uses CNNs for unconstrained character recognition.”
Gaussian Conditional Random Field Network for Semantic Segmentation (2016, methodology): “The idea of jointly training a CNN and graphical model has also been used for other applications such as sequence labeling [10, 34], text recognition [21], human pose estimation [52], predicting words from images [6], handwritten word recognition [4].”
Recursive Recurrent Nets with Attention Modeling for OCR in the Wild (2016, methodology): “[17] to prepare training and test sets that our method is trained purely on the Synth90k training set and all parameters are selected via validation set.”
Focusing Attention: Towards Accurate Text Recognition in Natural Images (2017, background): “designed a convolutional neural network (CNN) with structured output layer for unconstrained recognition [12].”
An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition (2015, methodology): “only the models based on deep neural networks including [6], [18] as well as CRNN have this property.”
Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition (2021, unknown): “2021) have been made to leverage a language model (LM) (Jaderberg et al. 2014a) to optimize the joint probability of the character sequence predicted by the VR model.”
Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes (2018, unknown): “[32] VGG, 90k 95.”
Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition (2021, background): “*The corresponding author methods [16, 14, 29] turn their attention to language modeling and achieve undoubted improvement.”
ASTER: An Attentional Scene Text Recognizer with Flexible Rectification (2019, background): “[25] proposed to construct a structuredoutput CNN for unconstrained text recognition.”
Reading Scene Text in Deep Convolutional Sequences (2015, methodology): “proposed another CNN model, which incorporates a Conditional Random Field (Jaderberg et al. 2015a).”