Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model

By Yixuan Qiao, Hao Chen, Jun Wang, Yihao Chen, Xia Ye, Ziliang Li, Xianbiao Qi, Peng Gao, G. Xie. 2021

9 citations recorded by the source corpus.

TextVQA requires models to read and reason about text in images to answer questions about them. Specifically, models need to incorporate a new modality of text present in the images and reason over it to answer TextVQA questions. In this challenge, we use generative model T5 for TextVQA task. Based on pre-trained checkpoint T5-3B from HuggingFace repository, two other pre-training tasks including masked language modeling(MLM) and relative position prediction(RPP) are designed to better align object feature and scene text. In the stage of pre-training, encoder is dedicate to handle the fusion among multiple modalities: question text, object text labels, scene text labels, object visual features, scene visual features. After that decoder generates the text sequence step-by-step, cross entropy loss is required by default. We use a large-scale scene text dataset in pre-training and then fine-tune the T5-3B with the TextVQA dataset only.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Jianfeng Gao
3. Mohit Bansal
4. Zicheng Liu
5. Lijuan Wang
6. Andrew Zisserman
7. Cordelia Schmid
8. Neil Houlsby

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Cordelia Schmid
3. Andrew Zisserman
4. Dacheng Tao
5. Trevor Darrell
6. Jitendra Malik
7. Ming-Hsuan Yang
8. Xiaogang Wang

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Linjie Li
6. Xiaowei Hu
7. Zhengyuan Yang
8. Jianfeng Gao

In Question Answering & Reading Comprehension

1. Radu Soricut
2. Soravit Changpinyo
3. Zhe Gan
4. Antoine Miech
5. Jianfeng Wang
6. Zicheng Liu
7. Xiaowei Hu
8. Zhengyuan Yang

Citations in the corpus

• Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “1 Restricted SotA† [140] [140] [124] [153] - - [51] [79] [79] [132] [137] [84] [62] 81.”
• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “(b): for TextVQA, Mia (Qiao et al., 2021)# is the winner entry of TextVQA Challenge 2021 with a fine-tuned T5-3B (Raffel et al.”
• PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “We fine-tune using OCR strings generated by publicly available automatic service, similar to the protocol in TAP (Yang et al., 2021) and Mia (Qiao et al., 2021).”
• PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, unknown): “5 - - - [5] [5] [52] [5] [5] [53] [54] [54] [45] [46] - - - PaLI-X 163.”