Unifying Multimodal Transformer for Bi-directional Image and Text Generation

By Yupan Huang, Hongwei Xue, Bei Liu, Yutong Lu, Yutong Lu. 2021

67 citations recorded by the source corpus.

We study the joint learning of image-to-text and text-to-image generations, which are naturally bi-directional tasks. Typical existing works design two separate task-specific models for each task, which impose expensive design efforts. In this work, we propose a unified image-and-text generative framework based on a single multimodal model to jointly study the bi-directional tasks. We adopt Transformer as our unified architecture for its strong performance and task-agnostic design. Specifically, we formulate both tasks as sequence generation tasks, where we represent images and text as unified sequences of tokens, and the Transformer learns multimodal interactions to generate sequences. We further propose two-level granularity feature representations and sequence-level training to improve the Transformer-based unified framework. Experiments show that our approach significantly improves previous Transformer-based model X-LXMERT's FID from 37.0 to 29.9 (lower is better) for text-to-image generation, and improves CIDEr-D score from 100.9% to 122.6% for fine-tuned image-to-text generation on the MS-COCO dataset. Our code is available online.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Jiebo Luo
2. Jianlong Fu
3. Houqiang Li
4. Bei Liu
5. Hongwei Xue
6. Dani Yogatama
7. Fangyu Liu
8. Lingpeng Kong

In Generative Models

1. Dani Yogatama
2. Hongwei Xue
3. Houqiang Li
4. Bei Liu
5. Jianlong Fu
6. Jiebo Luo
7. Lingpeng Kong
8. Yixuan Su

In Multimodal Learning

1. Hongwei Xue
2. Jiebo Luo
3. Bei Liu
4. Fangyu Liu
5. Tian Lan
6. Dani Yogatama
7. Houqiang Li
8. Yahui Liu

Citations in the corpus

• Language Models Can See: Plugging Visual Controls in Text Generation (2022, methodology): “After that, the main focus of following methods is to model the interaction between visual and textual cues via attention mechanism to get more faithful and richer captions [85, 46, 3, 91, 27].”
• CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, methodology): “…al., 2021; Xue et al., 2022; Zellers et al., 2021; Fu et al., 2021; Huang et al., 2020; 2021b; Xue et al., 2021; Li et al., 2021; Kim et al., 2021; Huang et al., 2021a; Sun et al., 2022) for vision-language pre-training are replacing the traditional approach using pre-extracted visual features by…”