Bilinear Graph Networks for Visual Question Answering

By Dalu Guo, Dacheng Tao. 2019

73 citations recorded by the source corpus.

This article revisits the bilinear attention networks (BANs) in the visual question answering task from a graph perspective. The classical BANs build a bilinear attention map to extract the joint representation of words in the question and objects in the image but lack fully exploring the relationship between words for complex reasoning. In contrast, we develop bilinear graph networks to model the context of the joint embeddings of words and objects. Two kinds of graphs are investigated, namely, image-graph and question-graph. The image-graph transfers features of the detected objects to their related query words, enabling the output nodes to have both semantic and factual information. The question-graph exchanges information between these output nodes from image-graph to amplify the implicit yet important relationship between objects. These two kinds of graphs cooperate with each other, and thus, our resulting model can build the relationship and dependency between objects, which leads to the realization of multistep reasoning. Experimental results on the VQA v2.0 validation dataset demonstrate the ability of our method to handle complex questions. On the test-std set, our best single model achieves state-of-the-art performance, boosting the overall accuracy to 72.56%, and we are one of the top-two entries in the VQA Challenge 2020.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Graph Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

Retrieval Augmented Visual Question Answering with Outside Knowledge (2022, background): “Models can aggregate visual and textual features via cross-modality fusion (Yu et al., 2018; Singh et al., 2019; Yu et al., 2019; Jiang et al., 2020; Guo et al., 2021).”
Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering (2022, background): “This task has been well studied in the research communities, and numerous cross-modal methods have achieved state-of-the-art performance [6,14,21,30,31, 34, 50, 63, 67, 69].”