Rosetta: Large Scale System for Text Detection and Recognition in Images

By Fedor Borisyuk, Albert Gordo, Viswanath Sivakumar. 2018

338 citations recorded by the source corpus.

In this paper we present a deployed, scalable optical character recognition (OCR) system, which we call Rosetta , designed to process images uploaded daily at Facebook scale. Sharing of image content has become one of the primary ways to communicate information among internet users within social networks such as Facebook, and the understanding of such media, including its textual information, is of paramount importance to facilitate search and recommendation applications. We present modeling techniques for efficient detection and recognition of text in images and describe Rosetta 's system architecture. We perform extensive evaluation of presented technologies, explain useful practical approaches to build an OCR system at scale, and provide insightful intuitions as to why and how certain components work based on the lessons learnt during the development and deployment of the system.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “In this study, we experiment with Rosetta and the Microsoft Azure OCR system (Microsoft-OCR).”
Towards VQA Models That Can Read (2019, methodology): “We also note that the drop in TextVQA for number of answers of a particular answer length is more gradual than in VQA 2.0 which drops sharply after answer length Finally, we analyze the OCR tokens produced by the Rosetta OCR system [6].”
Scene Text Visual Question Answering (2019, methodology): “Approaches that use Connectionist Temporal Classification have also been widely used in scene text recognition, in works such as [47, 7, 57, 12, 38], among others.”
FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents (2019, background): “Modern engines that also support handwritten text recognition usually use a Connectionist Temporal Classification (CTC) loss to cope with the alignment problem [18].”
Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications (2018, background): “One of the largest scale object detection systems running in data centers today is the text detection part of the Rosetta system used to understand text in images [8].”
Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems (2020, methodology): “Notice that the increase trend is not monotonic because in order to alleviate pressure on the training, inference and serving resources, we are constantly evaluating novel techniques to improve efﬁciency, such as quantization and compression.”
Confidence-aware Non-repetitive Multimodal Transformers for TextCaps (2020, neutral): “OCR system(s) # Total tokens # In GT tokens Rosetta-en 40.8k 5.5k Rosetta-en + CRAFT + ABCNet 117.5k 10.0k Table 4: OCR tokens analysis on validation set.”
DeepStore: In-Storage Acceleration for Intelligent Queries (2019, background): “Thanks to recent advancements in deep neural networks (DNNs) and the explosive increase and ubiquitous accessibility of data, deep learning has been enabling intelligent queries in versatile data retrieval applications to improve the quality of data services [20, 58, 59, 85].”
What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis (2019, methodology): “On the other hand, Rosetta [3] removed the BiLSTM to reduce computational complexity and memory consumption.”
Finding and Characterizing Information Warfare Campaigns (2020, background): “For social media image OCR, the state of the art is arguably the Facebook Rosetta system, a deep learning model that conducts OCR while taking into consideration the background as well [48].”
Scene Text Detection and Recognition: The Deep Learning Era (2018, background): “ing such scenario-speciﬁc APIs, including Baidu Inc., Tencent Inc. and Megvii Inc.. Recent development of fast and efﬁcient methods [119], [183] has also allowed the deployment of large-scale systems [10]. Companies including Google Inc. and Amazon Inc. are providing text extraction APIs. Despite th”
Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering (2020, methodology): “, 2019) model for question encoding, the Rosetta-en OCR system (Borisyuk et al., 2018) for OCR representation and a Faster-RCNN (Ren et al.”
Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text (2020, methodology): “Each question-answer pair comes along with a list of tokens extracted by Object Character Recognition (OCR) models, Rosetta [9].”
Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA (2019, methodology): “Finally, we extract text tokens on each image using the Rosetta OCR system [10].”