DVQA: Understanding Data Visualizations via Question Answering

By Kushal Kafle, Scott D. Cohen, Brian L. Price, Christopher Kanan. 2018

558 citations recorded by the source corpus.

Bar charts are an effective way to convey numeric information, but today's algorithms cannot parse them. Existing methods fail when faced with even minor variations in appearance. Here, we present DVQA, a dataset that tests many aspects of bar chart understanding in a question answering framework. Unlike visual question answering (VQA), DVQA requires processing words and answers that are unique to a particular bar chart. State-of-the-art VQA algorithms perform poorly on DVQA, and we propose two strong baselines that perform considerably better. Our work will enable algorithms to automatically extract numeric and semantic information from vast quantities of bar charts found in scientific publications, Internet articles, business reports, and many other areas.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Towards VQA Models That Can Read (2019, background): “DVQA [20] assesses automatic bar-chart understanding by training models to answer questions about graphs and plots.”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Current science problem datasets such as AI2D [16], DVQA [15], VLQA [51], and FOODWEDS [23] have contributed to multimodal reasoning in the scientiﬁc domain.”
REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, background): “It would be interesting to extend this to streaming chart question answering [37,38,40], object detection, visual query detection [2], and other problems in vision and language [39].”
Scene Text Visual Question Answering (2019, methodology): “Also related to the task proposed in this paper are the recent works of Kaﬂe et al. [20] and Kahou et al. [21] on question answering for bar charts and diagrams, and the work of Kembhavi et al. [25] on textbook question answering.”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “In recent years, there has been growing interest in multi-modal mathematical reasoning, which involves using multiple sources of information, such as text, tables, natural images, and diagrams, to solve mathematical problems (Kahou et al., 2017; Kafle et al., 2018; Lu et al., 2021b, 2022b).”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…[187], FigureQA (en) [105], VisText (en) [226], LRV-Instruction (en) [148], ArxivQA (en) [132], TabMWP (en) [166], MMC-Inst (en) [149], DVQA (en) [104], UniChart (en) [182], SimChart9K (en) [263], Chart2Text (en) [191], FinTabNet (zh) [312], SciTSR (zh) [39], Chart Synthetic Chart2Markdown (en)…”
InfographicVQA (2021, methodology): “DVQA [25], Fig-ureQA [26], and LEAF-QA [11] datasets deal with VQA on charts.”
Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023, methodology): “We use a mixture of publicly available data for the VQA task which includes GQA (Hudson 2This task is to generate noun/phrase grounded captions (Peng et al., 2023). andManning, 2019), VGQA (Krishna et al., 2017), VQAv2 (Goyal et al., 2017), DVQA (Kafle et al., 2018), OCRVQA (Mishra et al., 2019) and”
Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, unknown): “In comparison, our proposed Vision-R1 demonstrates a substantial advantage in reasoning performance, effectively balancing CoT complexity and accuracy.”
DocVQA: A Dataset for VQA on Document Images (2020, background): “[20, 19] comprises synthetically generated images of bar charts and template questions defined automatically based on the bar chart metadata.”
Cascade Reasoning Network for Text-based Visual Question Answering (2020, methodology): “Texts in such images generally convey valuable information and thus are of critical importance in visual understanding tasks such as visual question answering (VQA) [21].”
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models (2024, background): “…from 165k images by prompting 21], Infograph-icVQA train [22], AI2D train (transparent and opaque label boxes) [13], A-OKVQA train [28], Android-Control train [14], ScienceQA train [16], TabMWP train [17], ST-VQA train [6], TallyQA train [3], DVQA train [11], FigureQA train [12], and train [23].”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, unknown): “…QA VSR (en) [55], VisualDialog (en) [24] Science AI2D (en) [39], ScienceQA (en) [69], TQA (en) [40] ChartQA (en) [77], MMC-Inst (en) [57], DVQA (en) [38], Chart PlotQA (en) [81], LRV-Instruction (en) [56] GeoQA+ (en) [14], TabMWP (en) [70], MathQA (en) [127], Mathematics CLEVR-Math/Super…”
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (2022, methodology): “3M DVQA (Kafle et al., 2018) Template-based Fixed ✗ ✗ 300K/3.”
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning (2022, unknown): “Some recent datasets like DVQA (Kaﬂe et al., 2018), IconQA (Lu et al., 2021b), Geometry3K (Lu et al., 2021a), and UniGeo (Chen et al., 2022) introduce math problems with diagrams as the visual context, where the system needs to perform mathematical reasoning over multi-modal information.”
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning (2021, background): “Unlike natural images (VQA [3], CLEVR [21]) or abstract diagrams like scenes, charts, plots, and illustrations (VQA-Abstract [3], DVQA [22], NLVR [48], AI2D [26], Geometry3K [37]), IconQA features icon images and covers the largest object set of 388 classes.”
Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models (2024, unknown): “Building upon these advancements, several models [42, 8] have focused on generating high-quality instructional data based on BLIP-2, leading to marked improvements in performance.”