An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
By Cheng Zhang, Wei-Lun Chao, Dong Xuan. 2019
58 citations recorded by the source corpus.
Visual question answering (Visual QA) has attracted significant attention these years. While a variety of algorithms have been proposed, most of them are built upon different combinations of image and language features as well as multi-modal attention and fusion. In this paper, we investigate an alternative approach inspired by conventional QA systems that operate on knowledge graphs. Specifically, we investigate the use of scene graphs derived from images for Visual QA: an image is abstractly represented by a graph with nodes corresponding to object entities and edges to object relationships. We adapt the recently proposed graph network (GN) to encode the scene graph and perform structured reasoning according to the input question. Our empirical studies demonstrate that scene graphs can already capture essential information of images and graph networks have the potential to outperform state-of-the-art Visual QA algorithms but with a much cleaner architecture. By analyzing the features generated by GNs we can further interpret the reasoning process, suggesting a promising direction towards explainable Visual QA.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Aaron C. Courville
- Xiaojun Chang
- Graham W. Taylor
- Yoshua Bengio
- Alexander G. Hauptmann
- Trevor Darrell
- Zhihui Li
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Yi Yang
- Andrea Vedaldi
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
In Multimodal Learning
- Aaron C. Courville
- Harm de Vries
- Graham W. Taylor
- Xiaojun Chang
- Alexander G. Hauptmann
- Kate Saenko
- Kyunghyun Cho
- Trevor Darrell
Citations in the corpus
- Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation (2020, methodology): “This is essential for such tasks as visual question answering (VQA) [2, 16, 5] or caption generation [39, 12].”
- A Comprehensive Survey of Scene Graphs: Generation and Application (2021, methodology): “Inspired by the application of traditional QA systems on knowledge graphs, an alternative scene graph-based approach was investigated [175].”