A Diagram is Worth a Dozen Images

By Aniruddha Kembhavi, M. Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi. 2016

1,022 citations recorded by the source corpus.

Diagrams are common tools for representing complex concepts, relationships and events, often when it would be difficult to portray the same information with natural images. Understanding natural images has been extensively studied in computer vision, while diagram understanding has received little attention. In this paper, we study the problem of diagram interpretation, the challenging task of identifying the structure of a diagram and the semantics of its constituents and their relationships. We introduce Diagram Parse Graphs (DPG) as our representation to model the structure of diagrams. We define syntactic parsing of diagrams as learning to infer DPGs for diagrams and study semantic interpretation and reasoning of diagrams in the context of diagram question answering. We devise an LSTM-based method for syntactic parsing of diagrams and introduce a DPG-based attention model for diagram question answering. We compile a new dataset of diagrams with exhaustive annotations of constituents and relationships for about 5,000 diagrams and 15,000 questions and answers. Our results show the significance of our models for syntactic parsing and question answering in diagrams using DPGs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Hannaneh Hajishirzi
  6. Yejin Choi
  7. Ali Farhadi
  8. Christopher D. Manning

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Xiaogang Wang
  4. Trevor Darrell
  5. Chunhua Shen
  6. Ming-Hsuan Yang
  7. Jitendra Malik
  8. Alan L. Yuille

In Multimodal Learning

  1. Anton van den Hengel
  2. Damien Teney
  3. Qi Wu
  4. Christopher Kanan
  5. Kushal Kafle
  6. Peng Wang
  7. Chunhua Shen
  8. Devi Parikh

In Question Answering & Reading Comprehension

  1. Christopher Kanan
  2. Kushal Kafle
  3. Anton van den Hengel
  4. Damien Teney
  5. Robik Shrestha
  6. Qi Wu
  7. Chunhua Shen
  8. Scott D. Cohen

Citations in the corpus