Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text

By Difei Gao, Ke Li, Ruiping Wang, Shiguang Shan, Xilin Chen. 2020

132 citations recorded by the source corpus.

Answering questions that require reading texts in an image is challenging for current models. One key difficulty of this task is that rare, polysemous, and ambiguous words frequently appear in images, e.g., names of places, products, and sports teams. To overcome this difficulty, only resorting to pre-trained word embedding models is far from enough. A desired model should utilize the rich information in multiple modalities of the image to help understand the meaning of scene texts, e.g., the prominent text on a bottle is most likely to be the brand. Following this idea, we propose a novel VQA approach, Multi-Modal Graph Neural Network (MM-GNN). It first represents an image as a graph consisting of three sub-graphs, depicting visual, semantic, and numeric modalities respectively. Then, we introduce three aggregators which guide the message passing from one graph to another to utilize the contexts in various modalities, so as to refine the features of nodes. The updated nodes have better features for the downstream question answering module. Experimental evaluations show that our MM-GNN represents the scene texts better and obviously facilitates the performances on two VQA tasks that require reading scene texts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lijuan Wang
2. Jianfeng Wang
3. Qi Wu
4. Anton van den Hengel
5. Zhengyuan Yang
6. Zhe Gan
7. Peng Wang
8. Jiebo Luo

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Chunhua Shen
4. Andrew Zisserman
5. Trevor Darrell
6. Xiaogang Wang
7. Cordelia Schmid
8. Ming-Hsuan Yang

In Multimodal Learning

1. Jianfeng Wang
2. Lijuan Wang
3. Zhengyuan Yang
4. Qi Wu
5. Anton van den Hengel
6. Peng Wang
7. Zhe Gan
8. Jiebo Luo

In Graph Learning

1. Qi Wu
2. Jiebo Luo
3. Lei Zhang
4. Anton van den Hengel

Citations in the corpus

• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “Recent studies [49, 17, 16, 20, 25, 55, 36, 19] proposed various network architectures to improve the Text-VQA/Text-Caption performance.”
• Structured Multimodal Attentions for TextVQA (2020, methodology): “MM-GNN [20] focused on construct visual graph, semantic graph and numeric graph for representing the information of scene texts.”
• LaTr: Layout-Aware Transformer for Scene-Text VQA (2021, methodology): “MM-GNN [15] builds separate graphs for different modalities by utilizing graph neural networks [27].”
• DocVQA: A Dataset for VQA on Document Images (2020, methodology): “The ST-VQA [5] and TextVQA [35] datasets were introduced in parallel in 2019 and were quickly followed by more research [36, 11, 39].”
• Deep Multimodal Data Fusion (2024, background): “…based on GNNs addressing the multimodal problems related to graph data, e.g., VQA task [77, 95, 100, 129, 190, 214, 218], image captioning task [42, 182, 182, 208–210], cross-modal retrieval task [30, 204, 217], RGB-depth scene classification task [222], multimodal recommendation task [193,…”