PlotQA: Reasoning over Scientific Plots

By Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, Pratyush Kumar. IEEE Workshop/Winter Conference on Applications of Computer Vision, 2019

449 citations recorded by the source corpus.

Existing synthetic datasets (Figure QA, DVQA) for reasoning over plots do not contain variability in data labels, real-valued data, or complex reasoning questions. Consequently, proposed models for these datasets do not fully address the challenge of reasoning over plots. In particular, they assume that the answer comes either from a small fixed size vocabulary or from a bounding box within the image. However, in practice, this is an unrealistic assumption because many questions require reasoning and thus have real-valued answers which appear neither in a small fixed size vocabulary nor in the image. In this work, we aim to bridge this gap between existing datasets and real-world plots. Specifically, we propose PlotQA with 28.9 million question-answer pairs over 224,377 plots on data from real-world sources and questions based on crowd-sourced question templates. Further, 80.76% of the out-of-vocabulary (OOV) questions in PlotQA have answers that are not in a fixed vocabulary. Analysis of existing models on PlotQA reveals that they cannot deal with OOV questions: their overall accuracy on our dataset is in single digits. This is not surprising given that these models were not designed for such questions. As a step towards a more holistic model which can address fixed vocabulary as well as OOV questions, we propose a hybrid approach: Specific questions are answered by choosing the answer from a fixed vocabulary or by extracting it from a predicted bounding box in the plot, while other questions are answered with a table question-answering engine which is fed with a structured table generated by detecting visual elements from the image. On the existing DVQA dataset, our model has an accuracy of 58%, significantly improving on the highest reported accuracy of 46%. On PlotQA, our model has an accuracy of 22.52%, which is significantly better than state of the art models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c496f00f2b85b496abdc36e4dc4105541248fd3c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, unknown): “In comparison, our proposed Vision-R1 demonstrates a substantial advantage in reasoning performance, effectively balancing CoT complexity and accuracy.”
MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding (2021, unknown): “Several datasets have been proposed, such as VQA (Agrawal et al. 2017; Goyal et al. 2017), DAQUAR (Malinowski and Fritz 2014), COCO-QA (Ren, Kiros, and Zemel 2015), VCR (Zellers et al. 2019), and PlotQA (Methani et al. 2020), all of which require answering questions about images.”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, unknown): “Chart interpretation is bolstered by ChartQA [77], MMC-Inst [57], and PlotQA [81], which train the model to analyze and understand chart images.”
What matters when building vision-language models? (2024, unknown): “General visual question answering VQAv2 (Goyal et al., 2017) 82,772 443,757 1,595,929 5.72% COCO-QA (Ren et al., 2015) 46,287 78,736 286,982 1.47% Visual7W (Zhu et al., 2016) 14,366 69,817 279,268 1.43% A-OKVQA (Schwenk et al., 2022) 16,539 17,056 236,492 1.21% TallyQA (Acharya et al., 2019) 98,680 ”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, unknown): “…[40], Docmatix [41] 1.31M Chart/Figure ChartQA [42], MMC_Instruction [83], DVQA [84], LRV_Instruction [85], ChartGemma [86], InfoVQA [87], PlotQA [88] 1.00M OCR MultiUI [89], in-house data 0.83M Grounding RefCoco [90], VCR [91], in-house data 0.50M Multi-Image Demon-Full [92],…”
MultiModalQA: Complex Question Answering over Text, Tables and Images (2021, unknown): “…of answering questions about images—has been widely explored in previous work (Antol et al., 2015; Zhang et al., 2016; Goyal et al., 2017; Johnson et al., 2017; Hudson & Manning, 2019; Zellers et al., 2019; Singh et al., 2019; Methani et al., 2020), ranging from synthetic images to scientiﬁc plots.”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…et al., 2021b; Ji et al., 2022), geometry diagrams (Seo et al., 2015; Lu et al., 2021a; Chen et al., 2022a; Cao & Xiao, 2022), figures and charts (Methani et al., 2020; Masry et al., 2022; Kahou et al., 2017; Chang et al., 2022; Kafle et al., 2018), documents (text in images) (Singh et al.,…”
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models (2024, background): “…from 165k images by prompting 21], Infograph-icVQA train [22], AI2D train (transparent and opaque label boxes) [13], A-OKVQA train [28], Android-Control train [14], ScienceQA train [16], TabMWP train [17], ST-VQA train [6], TallyQA train [3], DVQA train [11], FigureQA train [12], and train [23].”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…MetaMath-Rendered [281], MapQA [23], GeoQA+ [20], Geometry3K [164], Mathematics UniGeo [26], GEOS [206], CLEVR-Math [144] ChartQA [181], PlotQA [187], FigureQA [105], LRV-Instruction [148], ArxivQA [132], MMC-Inst [149], TabMWP [166], DVQA [104], UniChart [182], SimChart9K [263], Chart2Text…”
TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text (2021, background): “A series of datasets were introduced following these which focusing on specific aspects of text-based VQA including OCR-VQA [39], STE-VQA [60], DocVQA [36], PlotQA [37], and LEAF-QA [7].”
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (2022, methodology): “We evaluate our models on three datasets from previous work namely, FigureQA (Kahou et al., 2017), PlotQA (Methani et al., 2020) and DVQA (Kafle et al., 2018), as well as our newly created ChartQA dataset.”
MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation (2026, unknown): “Performance is measured using BERT Matching Score ( BEM ) [4] for E-VQA, and standard accuracy [21] or relaxed accuracy [56] for InfoSeek.”
mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQA (2025, unknown): “We adopt the dataset’s evaluation metrics: VQA Accuracy [17] for STRING and TIME questions, and Relaxed Accuracy [48] for NUMERICAL questions.”
jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval (2025, unknown): “For example, for DonutVQA 6 , TableVQA [Tom Agonnoude, 2024], MP-MQA [Zhang et al., 2023], CharXiv [Wang et al., 2024], and PlotQA [Methani et al., 2020], we used structured templates and generative language models to formulate text queries to match their contents.”