Answering Questions about Data Visualizations using Efficient Bimodal Fusion
By Kushal Kafle, Robik Shrestha, Brian L. Price, Scott D. Cohen, Christopher Kanan. 2019
67 citations recorded by the source corpus.
Chart question answering (CQA) is a newly proposed visual question answering (VQA) task where an algorithm must answer questions about data visualizations, e.g. bar charts, pie charts, and line graphs. CQA requires capabilities that natural-image VQA algorithms lack: fine-grained measurements, optical character recognition, and handling out-of-vocabulary words in both questions and answers. Without modifications, state-of-the-art VQA algorithms perform poorly on this task. Here, we propose a novel CQA algorithm called parallel recurrent fusion of image and language (PReFIL). PReFIL first learns bimodal embeddings by fusing question and image features and then intelligently aggregates these learned embeddings to answer the given question. Despite its simplicity, PReFIL greatly surpasses state-of-the art systems and human baselines on both the FigureQA and DVQA datasets. Additionally, we demonstrate that PReFIL can be used to reconstruct tables by asking a series of questions about a chart.
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Christopher Kanan
- Tyler L. Hayes
- Kushal Kafle
- Robik Shrestha
- Manoj Acharya
- Dimosthenis Karatzas
- C. V. Jawahar
- Shafiq R. Joty
Citations in the corpus
- REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, background): “It would be interesting to extend this to streaming chart question answering [37,38,40], object detection, visual query detection [2], and other problems in vision and language [39].”
- DocVQA: A Dataset for VQA on Document Images (2020, background): “[20, 19] comprises synthetically generated images of bar charts and template questions defined automatically based on the bar chart metadata.”
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (2022, methodology): “• PREFIL (Kafle et al., 2019) is a classification approach that fuses the question and image features in parallel.”