IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

By Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, Song-Chun Zhu. 2021

307 citations recorded by the source corpus.

Current visual question answering (VQA) tasks mainly consider answering human-annotated questions for natural images. However, aside from natural images, abstract diagrams with semantic richness are still understudied in visual understanding and reasoning research. In this work, we introduce a new challenge of Icon Question Answering (IconQA) with the goal of answering a question in an icon image context. We release IconQA, a large-scale dataset that consists of 107,439 questions and three sub-tasks: multi-image-choice, multi-text-choice, and filling-in-the-blank. The IconQA dataset is inspired by real-world diagram word problems that highlight the importance of abstract diagram understanding and comprehensive cognitive reasoning. Thus, IconQA requires not only perception skills like object recognition and text understanding, but also diverse cognitive reasoning skills, such as geometric reasoning, commonsense reasoning, and arithmetic reasoning. To facilitate potential IconQA models to learn semantic representations for icon images, we further release an icon dataset Icon645 which contains 645,687 colored icons on 377 classes. We conduct extensive user studies and blind experiments and reproduce a wide range of advanced VQA methods to benchmark the IconQA task. Also, we develop a strong IconQA baseline Patch-TRM that applies a pyramid cross-modal Transformer with input diagram embeddings pre-trained on the icon dataset. IconQA and Icon645 are available at https://iconqa.github.io.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “In Table 2, we compare LLaMA-Adapter with popular VQA methods [2, 9, 17, 18, 22, 23, 28, 53] and language models [5, 16, 59].”
Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “As the two-stage method (QCMR → A) in Table 3 achieves better performance than all the one-stage method in Table 2, we choose the two-stage method in our Multimodal-CoT framework.”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Our experiments show that current multimodal methods [54, 1, 20, 8, 25, 34] fail to achieve satisfactory performance on S CIENCE QA and do not generate correct explanations.”
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “Furthermore, we evaluate general VQA benchmarks, such as VQAV2 (Agrawal et al., 2015), OKVQA (Marino et al., 2019), GQA (Hudson & Manning, 2019), vizwiz (Gurari et al., 2018), ScienceQA (Lu et al., 2022), visual spatial reasoning (VSR) (Liu et al., 2023a), IconQA (Lu et al., 2021).”
ImageBind-LLM: Multi-modality Instruction Tuning (2023, methodology): “We employ 9 benchmarks in the VQA task, namely DocVQA [73], TextVQA [74], STVQA [75], OCR-VQA [76], OKVQA [77], GQA [78], IconQA [79], Visual Spatial Reasoning (VSR) [80], and Visual Dialog (Visdial) [81].”
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (2023, background): “…categories and 26 datasets, including image captioning [10, 11, 12], image captioning with reading comprehension [13], visual reasoning [14, 15, 16], image question answering [17, 18], knowledge-grounded image question answering [19, 20, 21], image question answering with reading comprehension…”
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models (2023, unknown): “Conclusion: This paper proposes a comprehensive evaluation benchmark for large vision-language models called LVLM-eHub that incorporates both quantitative performance evaluation and human feedback evaluation.”
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning (2022, unknown): “Some recent datasets like DVQA (Kaﬂe et al., 2018), IconQA (Lu et al., 2021b), Geometry3K (Lu et al., 2021a), and UniGeo (Chen et al., 2022) introduce math problems with diagrams as the visual context, where the system needs to perform mathematical reasoning over multi-modal information.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…Synthetic Infographic2Markdown KVQA [207], A-OKVQA [205], ViQuAE [123], iNaturalist2018 [237], MovieNet [95], ART500K [176], KonIQ-10K [91], IconQA [167], VisualMRC [225], ChemVLM Data [129], ScienceQA [165], AI2D [109], Knowledge TQA [110], Wikipedia-QA [81], Synthetic Multidisciplinary Knowledge…”
DeepSeek-VL: Towards Real-World Vision-Language Understanding (2024, unknown): “6-GPT4V (Liu et al., 2024a) and IconQA (Lu et al., 2021).”
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition (2023, unknown): “Caption COCO [8], SUB [8], TextCaps [59] VQA VQAv2 [1], GQA [26], OK-VQA [44], VSR [35], IConQA [42] Text-VQA [60], SQA [41], OCR-VQA [45], Visual Dialog [12], LLava-150k [38] Instruction tuning Image-Text Composiiton In-house data (Refer to Sec.3.3) Conversation LLaVA-150k [38], Alpaca-en&zh [61],…”
What matters when building vision-language models? (2024, unknown): “TabMWP (Lu et al., 2023) 22,729 23,059 1,948,166 2.49% TAT-QA (Zhu et al., 2021) 2,199 13,215 283,776 2.18% HiTab (Cheng et al., 2022) 2,500 7,782 351,299 1.80% MultiHiertt (Zhao et al., 2022) 7,619 7,830 267,615 1.37% FinQA (Chen et al., 2021) 5,276 6,251 242,561 0.99% WikiSQL (Zhong et al., 2017) ”
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, unknown): “, 2022), IconQA-text (Lu et al., 2021), VisDial (Das et al.”
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, unknown): “The evaluation on general visual question answering (VQA) benchmarks such as VQAV2 (Agrawal et al., 2015), GQV (Hudson & Man-ning, 2019), OK-VQA (Marino et al., 2019), VizWiz (Gurari et al., 2018), ScienceQA (Lu et al., 2022), IconQA (Lu et al., 2021b) are presented in Table 3.”
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “More advanced image-sequence understanding capabilities are required in practical applications, such as Multi-Image Reasoning (Suhr et al., 2018; Lu et al., 2021; Jiang et al., 2024), Multimodal RAG (Chen et al., 2022; Lin et al., 2024), Video Understanding (Xiao et al., 2021; Li et al., 2023c; Fu…”
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, background): “[32] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu.”
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (2023, methodology): “In order to gain a comprehensive understanding of the limitations of LLMs in scientific problem solving, we propose a novel self-refinement method to uncover the deficient skills in the solutions made by LLMs. Firstly, we compare the correct solutions with the solutions generated by LLMs and, with t”
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, unknown): “2 [124], which cover a wide range of domains, including general visual question answering (VQA) [43, 50, 90, 83, 127, 126], science [57, 16, 82], chart [91, 54, 11], mathematics [72, 104, 10, 81, 55, 40, 147, 106], OCR [92, 107, 9, 49, 96], and document [24].”