Multimodal Conditionality for Natural Language Generation
By Michael Sollami, Aashish Jain. 2021
10 citations recorded by the source corpus.
Large scale pretrained language models have demonstrated state-of-the-art performance in language understanding tasks. Their application has recently expanded into multimodality learning, leading to improved representations combining vision and language. However, progress in adapting language models towards conditional Natural Language Generation (NLG) has been limited to a single modality, generally text. We propose MAnTiS, Multimodal Adaptation for Text Synthesis, a general approach for multimodal conditionality in transformer-based NLG models. In this method, we pass inputs from each modality through modality-specific encoders, project to textual token space, and finally join to form a conditionality prefix. We fine-tune the pretrained language model and encoders with the conditionality prefix guiding the generation. We apply MAnTiS to the task of product description generation, conditioning a network on both product images and titles to generate descriptive text. We demonstrate that MAnTiS outperforms strong baseline approaches on standard NLG scoring metrics. Furthermore, qualitative assessments demonstrate that MAnTiS can generate human quality descriptions consistent with given multimodal inputs.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Constantin Eichenberg
- Anette Frank
- Jyoti Prakash Sahoo
- Yisheng Song
- Samuel Weinbach
- Sid Black
- Ting-Yuan Wang
- Subrota Kumar Mondal
In Computer Vision and Pattern Recognition
- Constantin Eichenberg
- Anette Frank
- Jyoti Prakash Sahoo
- Yisheng Song
- Samuel Weinbach
- Sid Black
- Ting-Yuan Wang
- Subrota Kumar Mondal
In Multimodal Learning
- Constantin Eichenberg
- Anette Frank
- Samuel Weinbach
- Sid Black
- Letitia Parcalabescu
- Ruslan Salakhutdinov
- Daniel Fried
- Jing Yu Koh
Citations in the corpus
- MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, background): “A recent line of work [47, 52, 55] explores VL modeling using autoregressive decoder models trained with a language modeling objective.”
- A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “The prompts of MCNLG [214] are very intuitive, which places multimodal sequences as prefix prompts in front of input sequences that decoder in turn the shared multimodal information.”