What BERT Sees: Cross-Modal Transfer for Visual Question Generation
By Thomas Scialom, Patrick Bordes, Paul-Alexis Dray, Jacopo Staiano, Patrick Gallinari. 2020
14 citations recorded by the source corpus.
Pre-trained language models have recently contributed to significant advances in NLP tasks. Recently, multi-modal versions of BERT have been developed, using heavy pre-training relying on vast corpora of aligned textual and image data, primarily applied to classification tasks such as VQA. In this paper, we are interested in evaluating the visual capabilities of BERT out-of-the-box, by avoiding pre-training made on supplementary data. We choose to study Visual Question Generation, a task of great interest for grounded dialog, that enables to study the impact of each modality (as input can be visual and/or textual). Moreover, the generation aspect of the task requires an adaptation since BERT is primarily designed as an encoder. We introduce BERT-gen, a BERT-based architecture for text generation, able to leverage on either mono- or multi- modal representations. The results reported under different configurations indicate an innate capacity for BERT-gen to adapt to multi-modal data and text generation, even with few data available, avoiding expensive pre-training. The proposed model obtains substantial improvements over the state-of-the-art on two established VQG datasets.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Felix Hill
- Mohit Bansal
- Jianfeng Gao
- Ruslan Salakhutdinov
- Quoc V. Le
- Yoshua Bengio
- Kyunghyun Cho
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Jitendra Malik
- Xiaogang Wang
- Andrew Zisserman
- Li Fei-Fei
- Ming-Hsuan Yang
In Multimodal Learning
- Felix Hill
- Oriol Vinyals
- Ruslan Salakhutdinov
- Marcus Rohrbach
- Devi Parikh
- Jianfeng Gao
- Qi Wu
- Quoc V. Le
Citations in the corpus
- Multimodal Few-Shot Learning with Frozen Language Models (2021, methodology): “[36] extend a similar image embedding+BERT system to create a generative model of text, using a pre-trained object extraction system to embed images into word space.”