MSMO: Multimodal Summarization with Multimodal Output

By Junnan Zhu, Haoran Li, Tianshan Liu, Yu Zhou, Jiajun Zhang, Chengqing Zong. 2018

196 citations recorded by the source corpus.

Multimodal summarization has drawn much attention due to the rapid growth of multimedia data. The output of the current multimodal summarization systems is usually represented in texts. However, we have found through experiments that multimodal output can significantly improve user satisfaction for informativeness of summaries. In this paper, we propose a novel task, multimodal summarization with multimodal output (MSMO). To handle this task, we first collect a large-scale dataset for MSMO research. We then propose a multimodal attention model to jointly generate text and select the most relevant image from the multimodal input. Finally, to evaluate multimodal outputs, we construct a novel multimodal automatic evaluation (MMAE) method which considers both intra-modality salience and inter-modality relevance. The experimental results show the effectiveness of MMAE.

Subfields: Artificial Intelligence, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Claire Cardie
2. Chenguang Zhu
3. Lu Yuan
4. Faisal Ladhak
5. Jiajun Zhang
6. Chengqing Zong
7. Junnan Zhu
8. Esin Durmus

In Natural Language Processing

1. Faisal Ladhak
2. Claire Cardie
3. K. McKeown
4. Esin Durmus
5. Junnan Zhu
6. Chengqing Zong
7. Jiajun Zhang
8. Chenguang Zhu

In Multimodal Learning

1. Lu Yuan
2. Claire Cardie
3. Bin Xiao
4. Yichong Xu
5. Chengqing Zong
6. Shaonan Wang
7. W. Zhang
8. Jiajun Zhang

Citations in the corpus

• WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, background): “Zhu et al. (2019) use round-trip translation of large scale monolingual datasets (Hermann et al., 2015; Zhu et al., 2018; Hu et al., 2015) to generate synthetic training data for their models, and train a multi-task model to to learn both translation and cross-lingual summarization.”
• Video Summarization Using Deep Neural Networks: A Survey (2021, background): “Addressing the video summarization task from a slightly different perspective, other multimodal algorithms generate a text-based summary of the video [69], [70], [71], and extend this output by extracting one representative key-frame [72].”
• i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, methodology): “Notably, i-Code V2 outperforms previous SOTA models on MSMO (multimodal summarization), Image Chat (multimodal dialogue generation), UR-FUNNY (multimodal sentiment analysis).”
• Multi-document Summarization via Deep Learning Techniques: A Survey (2020, background): “This kind of multi-modal summary can improve the satisfaction of users [158], (ii) text + video: Based on the video and its subtitles, generating a concise text summary that describes the main context of video [97].”
• NCLS: Neural Cross-Lingual Summarization (2019, background): “Specifically, we propose a simple yet effective round-trip translation strategy to obtain cross-lingual document-summary pairs from existing monolingual summarization datasets (Hermann et al., 2015; Zhu et al., 2018; Hu et al., 2015).”