X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

By Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, Aniruddha Kembhavi. 2020

109 citations recorded by the source corpus.

Mirroring the success of masked language models, vision-and-language counterparts like ViLBERT, LXMERT and UNITER have achieved state of the art performance on a variety of multimodal discriminative tasks like visual question answering and visual grounding. Recent work has also successfully adapted such models towards the generative task of image captioning. This begs the question: Can these models go the other way and generate images from pieces of text? Our analysis of a popular representative from this model family - LXMERT - finds that it is unable to generate rich and semantically meaningful imagery with its current training setup. We introduce X-LXMERT, an extension to LXMERT with training refinements including: discretizing visual representations, using uniform masking with a large range of masking ratios and aligning the right pre-training datasets to the right objectives which enables it to paint. X-LXMERT's image generation capabilities rival state of the art generative models while its question answering and captioning abilities remains comparable to LXMERT. Finally, we demonstrate the generality of these training refinements by adding image generation capabilities into UNITER to produce X-UNITER.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Zero-Shot Text-to-Image Generation (2021, methodology): “More recently, Cho et al. (2020) also propose a method that involves optimizing the input to a pretrained cross-modal masked language model.”
Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “Following this success, image+text pretraining models (Lu et al., 2019; Tan & Bansal, 2019; Chen et al., 2020; Huang et al., 2020; Li et al., 2020b; Cho et al., 2020; Radford et al., 2021; Zhang et al., 2021) and video+text pretraining models (Sun et al., 2019b;a; Li et al., 2020a; Zhu & Yang,…”
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, background): “…of vision and language, giving birth to the category of Vision-and-Language Pre-training (VLP) models (Lu et al., 2019; Chen et al., 2019; Su et al., 2019; Li et al., 2019; Tan & Bansal, 2019; Li et al., 2020a; Lu et al., 2020; Cho et al., 2020; Qi et al., 2020; Zhou et al., 2020; Huang Figure 1.”
Retrieval-Augmented Multimodal Language Modeling (2023, background): “Meanwhile, several works also study image-to-text generation (Cho et al., 2020; Wang et al., 2022b).”
Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture (2021, background): “Over the last decade, specialized and effective approaches have been developed for vision-language tasks, including image captioning [10,21,25,33,47,54], phrase grounding [37,38,43], referring expression comprehension [22, 34], visual question answering (VQA) [2,11,16,48,51,55], visual dialog [7], a”
M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers (2021, methodology): “During inference, we adopt Mask-Predict, a NAR generation algorithm [16, 21, 7], which predicts all target tokens at the first iteration and then iteratively re-mask and re-predict a subset of tokens with low confidence scores.”
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers (2022, methodology): “X-LXMERT [13] introduced a new language model approach by encoding an image as a grid of latent code and training a multimodal transformer language model [54] to learn the distribution of the image code sequence given a text input.”
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking (2022, background): “SelfDoc proposes to regress masked region features [29], which is noisier and harder to learn than classifying discrete features in a smaller vocabulary [6, 17].”
WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training (2021, background): “Most existing multi-modal pre-training models, especially those with the single-tower architecture [20, 35, 26, 38, 9, 11, 40, 22, 7, 14], take an assumption that there exists strong semantic correlation between the input imagetext pair.”
DALL-EVAL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models (2022, background): “X-LXMERT [14] and DALL-E [45] introduce multimodal transformer language models that learn the distribution of the sequence of discrete image codes given text input.”
Unifying Multimodal Transformer for Bi-directional Image and Text Generation (2021, methodology): “(1) Task-specific uni-directional architectures [2, 6, 53] on bi-directional image and text generation tasks.”
Benchmark for Compositional Text-to-Image Synthesis (2021, background): “Recent works explored a transformer-based vision-language representation model [5], cross-modal constrastive learning [59], and transformer and discrete VAE [39] for text-to-image synthesis.”
UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis (2021, methodology): “During inference, we adopt Mask-Predict, a NAR generation algorithm [16, 21, 7], which predicts all target tokens at the first iteration and then iteratively re-mask and re-predict a subset of tokens with low confidence scores.”