FigureQA: An Annotated Figure Dataset for Visual Reasoning
By Samira Ebrahimi Kahou, Adam Atkinson, Vincent Michalski, Ákos Kádár, Adam Trischler, Yoshua Bengio. 2017
488 citations recorded by the source corpus.
We introduce FigureQA, a visual reasoning corpus of over one million question-answer pairs grounded in over 100,000 images. The images are synthetic, scientific-style figures from five classes: line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts. We formulate our reasoning task by generating questions from 15 templates; questions concern various relationships between plot elements and examine characteristics like the maximum, the minimum, area-under-the-curve, smoothness, and intersection. To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure. To facilitate the training of machine learning systems, the corpus also includes side data that can be used to formulate auxiliary objectives. In particular, we provide the numerical data used to generate each figure as well as bounding-box annotations for all plot elements. We study the proposed visual reasoning task by training several models, including the recently proposed Relation Network as a strong baseline. Preliminary results indicate that the task poses a significant machine learning challenge. We envision FigureQA as a first step towards developing models that can intuitively recognize patterns from visual representations of data.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Cordelia Schmid
- Ming-Hsuan Yang
- Jitendra Malik
- Chunhua Shen
In Multimodal Learning
- Christopher Kanan
- Kushal Kafle
- Anton van den Hengel
- Qi Wu
- Devi Parikh
- Pan Lu
- Peng Wang
- Robik Shrestha
In Evaluation & Benchmarks
- Anton van den Hengel
- Mitesh M. Khapra
- Noah A. Smith
- Hannaneh Hajishirzi
- Mohit Bansal
- Alane Suhr
- Yoav Artzi
- Shafiq R. Joty
In Question Answering & Reading Comprehension
- Christopher Kanan
- Kushal Kafle
- Manoj Acharya
- Robik Shrestha
- Scott D. Cohen
- Brian L. Price
- Anton van den Hengel
- Qi Wu
Citations in the corpus
- A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “Several recent datasets focus on compositional reasoning about images and language, mostly using synthetic data for language and vision (Andreas et al., 2016b; Johnson et al., 2017a; Kuhnle and Copes-take, 2017; Kahou et al., 2018; Yang et al., 2018).”
- REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, background): “It would be interesting to extend this to streaming chart question answering [37, 38, 40], object detection, visual query detection [2], and other problems in vision and language [39].”
- Scene Text Visual Question Answering (2019, methodology): “Also related to the task proposed in this paper are the recent works of Kafle et al. [20] and Kahou et al. [21] on question answering for bar charts and diagrams, and the work of Kembhavi et al. [25] on textbook question answering.”
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (2022, methodology): “We evaluate our models on three datasets from previous work namely, FigureQA (Kahou et al., 2017), PlotQA (Methani et al., 2020) and DVQA (Kafle et al., 2018), as well as our newly created ChartQA dataset.”
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “However, current multimodal reasoning benchmarks provide limited coverage of rigorous and scientific domains (Antol et al., 2015; Kembhavi et al., 2016; Kahou et al., 2017; Mathew et al., 2022), which are key components for creating general-purpose AI assistants.”
- RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes (2018, methodology): “Among previous efforts towards multimodal machine comprehension (Tapaswi et al., 2016; Kemb-havi et al., 2016; Iyyer et al., 2017; Kembhavi et al., 2017; Kahou et al., 2018), our study is closer to what Kembhavi et al. (2017) envisioned in TQA.”
- What matters when building vision-language models? (2024, unknown): “…2, (Tang et al., 2023) 7,057 9,969 1,245,485 1.91% ChartQA (Masry et al., 2022) 18,271 28,299 185,835 1.90% PlotQA (Methani et al., 2020) 157,070 20,249,479 8478299.278 0.65% FigureQA (Kahou et al., 2017) 100,000 1,327,368 3,982,104 0.61% MapQA (Chang et al., 2022) 37,417 483,416 6,470,485 0.33%”
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…[281], MapQA [23], GeoQA+ [20], Geometry3K [164], Mathematics UniGeo [26], GEOS [206], CLEVR-Math [144] ChartQA [181], PlotQA [187], FigureQA [105], LRV-Instruction [148], ArxivQA [132], MMC-Inst [149], TabMWP [166], DVQA [104], UniChart [182], SimChart9K [263], Chart2Text [191], FinTabNet…”
- PlotQA: Reasoning over Scientific Plots (2019, background): “In the FigureQA dataset [13], all questions are binary wherein answers are either Yes or No, (see Figure 1a for an example).”
- Answering Questions about Data Visualizations using Efficient Bimodal Fusion (2019, methodology): “CQA requires capabilities not tested by other VQA tasks due to the innate differences in how information is presented in data visualizations [15, 18].”
- DocVQA: A Dataset for VQA on Document Images (2020, background): “FigureQA [21] comprises over one million yes or no questions, grounded on over 100,000 images.”
- IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning (2021, background): “Similarly, NLVR [48], FigureQA [23], and DVQA [22] feature diagrams of scientific plots that are generated with several figure types and question templates.”
- Structured Multimodal Attentions for TextVQA (2020, methodology): “eQA [13] have been introduced to study visual reasoning purely, without the consideration of OCR tokens; Wang et al.”
- DVQA: Understanding Data Visualizations via Question Answering (2018, methodology): “Concurrent to our work, FigureQA [18] also explores question answering for charts, however, with following major limitations compared to our DVQA dataset: 1) it contains only yes/no type questions; 2) it does not contain questions that require numeric values as answers; 3) it has fixed labels for ba”
- InfographicVQA (2021, background): “DVQA [25], Fig-ureQA [26], and LEAF-QA [11] datasets deal with VQA on charts.”
- Interpretable Visual Question Answering by Reasoning on Dependency Trees (2018, unknown): “The baseline methods “Text only”, “CNN+LSTM”, “CNN+LSTM on VGG-16 features” and “RN [45]” were originally reported in [42].”
- A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “, 2022), generated from templates (Kahou et al., 2017), or focus on specific topics (Lu et al.”