On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering

By Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, C. Chan, Anton van den Hengel, Liangwei Wang. 2020

128 citations recorded by the source corpus.

Visual Question Answering (VQA) methods have made incredible progress, but suffer from a failure to generalize. This is visible in the fact that they are vulnerable to learning coincidental correlations in the data rather than deeper relations between image content and ideas expressed in language. We present a dataset that takes a step towards addressing this problem in that it contains questions expressed in two languages, and an evaluation process that co-opts a well understood image-based metric to reflect the method’s ability to reason. Measuring reasoning directly encourages generalization by penalizing answers that are coincidentally correct. The dataset reflects the scene-text version of the VQA problem, and the reasoning evaluation can be seen as a text-based version of a referring expression challenge. Experiments and analyses are provided that show the value of the dataset. The dataset is available at www.est-vqa.org.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “…[96] VQAv2 [40], OKVQA [79], A-OKVQA [93], VQA 1.1M IconQA [75], AI2D [53], GQA [48] OCR-VQA [82], ChartQA [80], DocVQA [25], ST-VQA [11], EST-VQA [116], InfoVQA [81] model can generate a feature map F ∈ R H/ 14 × W/ 14 × D for dense prediction tasks, or work with global average pooling and…”
Monkey: Image Resolution and Text Label are Important Things for Large Multi-Modal Models (2023, unknown): “For our evaluation, we employ four datasets: TextVQA [45], AI2D [23], STVQA [5], and ESTVQA [49].”
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning (2021, background): “Several datasets have been released to evaluate the systems’ visual and textual content understanding abilities [3, 57, 14, 21, 18, 52].”
Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering (2020, background): “2 Evidence-based VQA and Multitask Learning Evidence-based VQA has been proposed in the recent work (Wang et al., 2020), which suggests to use intersection over union (IoU) to indicate the evidence.”
OCRBench: on the hidden mystery of OCR in large multimodal models (2023, unknown): “Scene Text-Centric VQA : We test LMMs on STVQA [51], TextVQA [52], OCRVQA [53] and ESTVQA [54].”
LaTr: Layout-Aware Transformer for Scene-Text VQA (2021, background): “Moreover, STE-VQA [67] is proposed for multi-lingual VQA and TextCaps [55] for captioning on natural images.”
DocVQA: A Dataset for VQA on Document Images (2020, methodology): “The ST-VQA [5] and TextVQA [35] datasets were introduced in parallel in 2019 and were quickly followed by more research [36, 11, 39].”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…[75], ParsynthOCR [89], SynthDoG-EN [112], SynthDoG-ZH [112], SynthDoG-RU [112], SynthDoG-JP [112], SynthDoG-KO [112], IAM [180], EST-VQA [253], ST-VQA [17], NAF [52], InfoVQA [183], HME100K [288], OCRVQA [188], SROIE [97], POIE [115], CTW [287], SynthText [79], ArT [40], LSVT [222],…”
TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text (2021, background): “A series of datasets were introduced following these which focusing on specific aspects of text-based VQA including OCR-VQA [39], STE-VQA [60], DocVQA [36], PlotQA [37], and LEAF-QA [7].”