Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization

By Tiezheng Yu, Wenliang Dai, Zihan Liu, Pascale Fung. 2021

85 citations recorded by the source corpus.

Multimodal abstractive summarization (MAS) models that summarize videos (vision modality) and their corresponding transcripts (text modality) are able to extract the essential information from massive multimodal data on the Internet. Recently, large-scale generative pre-trained language models (GPLMs) have been shown to be effective in text generation tasks. However, existing MAS models cannot leverage GPLMs’ powerful generation ability. To fill this research gap, we aim to study two research questions: 1) how to inject visual information into GPLMs without hurting their generation ability; and 2) where is the optimal place in GPLMs to inject the visual information? In this paper, we present a simple yet effective method to construct vision guided (VG) GPLMs for the MAS task using attention-based add-on layers to incorporate visual information while maintaining their original text generation ability. Results show that our best model significantly surpasses the prior state-of-the-art model by 5.7 ROUGE-1, 5.3 ROUGE-2, and 5.1 ROUGE-L scores on the How2 dataset, and our vision guidance method contributes 83.6% of the overall improvement. Furthermore, we conduct thorough ablation studies to analyze the effectiveness of various modality fusion methods and fusion locations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pascale Fung
2. Wenliang Dai
3. Qun Liu
4. Xin Jiang
5. Lifeng Shang
6. Lu Hou
7. Dan Su
8. Zihan Liu

In Computer Vision and Pattern Recognition

1. Qun Liu
2. Wenliang Dai
3. Xin Jiang
4. Lifeng Shang
5. Lu Hou
6. Pascale Fung
7. Ziwei Ji
8. Dan Su

In Multimodal Learning

1. Wenliang Dai
2. Pascale Fung
3. Dan Su
4. Ziwei Ji
5. Zihan Liu
6. Tiezheng Yu
7. Qun Liu
8. Lifeng Shang

In Summarization

1. Zihan Liu
2. Tiezheng Yu
3. Pascale Fung

Citations in the corpus

• Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, methodology): “Additionally, as presented in Table 5, we also run VLKD on the abstractive summarization task to evaluate its NLG performance, since BART-based methods excel on the summarization (Lewis et al., 2020; Dou et al., 2021; Yu et al., 2021b).”
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, background): “…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
• Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2022, methodology): “Later, encoder-decoder architectures are introduced to additionally handle multimodal generation tasks with a causal language modeling loss (Li et al., 2021b; Yu et al., 2021b; Lin et al., 2021; Cho et al., 2021; Ding et al., 2021; Li et al., 2022; Wang et al., 2022a).”