A Diagram is Worth a Dozen Images
By Aniruddha Kembhavi, M. Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi. 2016
1,022 citations recorded by the source corpus.
Diagrams are common tools for representing complex concepts, relationships and events, often when it would be difficult to portray the same information with natural images. Understanding natural images has been extensively studied in computer vision, while diagram understanding has received little attention. In this paper, we study the problem of diagram interpretation, the challenging task of identifying the structure of a diagram and the semantics of its constituents and their relationships. We introduce Diagram Parse Graphs (DPG) as our representation to model the structure of diagrams. We define syntactic parsing of diagrams as learning to infer DPGs for diagrams and study semantic interpretation and reasoning of diagrams in the context of diagram question answering. We devise an LSTM-based method for syntactic parsing of diagrams and introduce a DPG-based attention model for diagram question answering. We compile a new dataset of diagrams with exhaustive annotations of constituents and relationships for about 5,000 diagrams and 15,000 questions and answers. Our results show the significance of our models for syntactic parsing and question answering in diagrams using DPGs.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Hannaneh Hajishirzi
- Yejin Choi
- Ali Farhadi
- Christopher D. Manning
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Chunhua Shen
- Ming-Hsuan Yang
- Jitendra Malik
- Alan L. Yuille
In Multimodal Learning
- Anton van den Hengel
- Damien Teney
- Qi Wu
- Christopher Kanan
- Kushal Kafle
- Peng Wang
- Chunhua Shen
- Devi Parikh
In Question Answering & Reading Comprehension
- Christopher Kanan
- Kushal Kafle
- Anton van den Hengel
- Damien Teney
- Robik Shrestha
- Qi Wu
- Chunhua Shen
- Scott D. Cohen
Citations in the corpus
- Towards VQA Models That Can Read (2019, neutral): “Similarly, AI2D [23] contains diagram based multiple-choice questions.”
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Different from existing datasets [17, 18, 52], SCIENCEQA has richer domain diversity from three different subjects: natural science, social science, and language science.”
- IQA: Visual Question Answering in Interactive Environments (2017, background): “These include datasets of natural images [2, 36, 41, 68], synthetic images [2, 4, 24, 27, 28], natural videos [23, 65], synthetic videos [32, 49] and multimodal contexts [29].”
- Semantic Parsing to Probabilistic Programs for Situated Question Answering (2016, methodology): “We preprocess the diagrams in the data set using a computer vision system that identifies candidate diagram elements (Kembhavi et al., 2016).”
- The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers (2018, background): “In [93], the problem of visual understanding is addressed in the context of science diagrams.”
- LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models (2024, neutral): “To evaluate the potential contamination of current benchmarks, we selected over 20 benchmarks, including AI2D [29], ChartQA [54], NoCaps [1], VQA v2 [21], and LLaVA-in-the-wild [42].”
- Gemma 3 Technical Report (2025, neutral): “Gemma 3 also performs better on AI2D and OKVQA, but PaliGemma 2 performs slightly better on VQAv2 and COCO caption.”
- Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models (2024, neutral): “To augment the data quality, we collect and produce more data based on public resources, including high-quality responses [14, 15], task-oriented instructions [16–19], and generation-related data [20, 21].”
- Qwen3-VL Technical Report (2025, neutral): “On OCR-related visual question answering (VQA) benchmarks that require both OCR capability and keyword search — such as DocVQA (Mathew et al., 2021b), InfoVQA (Mathew et al., 2021a), AI2D (Kembhavi et al., 2016), ChartQA (Masry et al., 2022), and the CharXiv (Wang et al., 2024g) description subset —”
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, neutral): “…VQA capabilities, we incorporate datasets including TextVQA (Singh et al., 2019), DocVQA (Mathew et al., 2021b), ChartQA (Masry et al., 2022), AI2D (Kembhavi et al., 2016), Deep-form (Svetlichnaya, 2020), DVQA (Kafle et al., 2018), InfographicsVQ (Mathew et al., 2021a), KleisterChar-ity…”
- Qwen2.5-Omni Technical Report (2025, neutral): “5-Omni on various OCR benchmarks, such as AI2D (Kembhavi et al., 2016), TextVQA (Singh et al., 2019), DocVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022), and OCRBench_v2 (Fu et al., 2024b).”
- Long Context Transfer from Language to Vision (2024, methodology): “Model AI2D [32] ChartQA [52] DocVQA [53] InfoVQA [53] RealworldQA [75] MMMU [86] LLaVA-1 We further evaluate our model on various image benchmarks to investigate the image performance of LongVA (Table 7).”
- Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023, methodology): “We assess our models’ ability toward text-oriented visual question answering on several benchmarks includingTextVQA(Sidorov et al., 2020), DocVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022), AI2Diagram (Kembhavi et al., 2016), and OCR-VQA (Mishra et al., 2019).”
- The Claude 3 Model Family: Opus, Sonnet, Haiku (background): “A prime example is the models’ performance on the AI2D science diagram benchmark [52], a visual question answering evaluation that involves diagram parsing and answering corresponding questions in a multiple-choice format.”
- Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “…task #samples dataset Captioning 588K COCO Caption [20], TextCaps [96] VQAv2 [40], OKVQA [79], A-OKVQA [93], VQA 1.1M IconQA [75], AI2D [53], GQA [48] OCR-VQA [82], ChartQA [80], DocVQA [25], ST-VQA [11], EST-VQA [116], InfoVQA [81] model can generate a feature map F ∈ R H/ 14 × W/ 14…”
- LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, neutral): “ShareGPT4V [9] General VQA 31.3k ChartQA [40] General VQA 17.2k A-OKVQA [48] General VQA 16.1k AI2D [25] Science-Targeted VQA 11.4k GeoQA+ [7] Science-Targeted VQA 11.4k ScienceQA [36] Science-Targeted VQA 5.6k DocVQA [41] General VQA 4.0k PISC [30] General VQA 1.0k CLEVR [24] General VQA 0.5k…”
- VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, neutral): “The benchmarks include: 1) RealWorldQA [120], which focuses on answering questions based on realistic images drawn from everyday scenarios, 2) AI2D [121], which evaluates the model’s ability to reason about diagrams and science images, 3) GQA [122], which assesses general question answering with a…”