XGPT: Cross-modal Generative Pre-Training for Image Captioning

By Qiaolin Xia, Haoyang Huang, Nan Duan, Dongdong Zhang, Lei Ji, Zhifang Sui, Edward Cui, Taroon Bharti, Xin Liu, Ming Zhou. 2020

84 citations recorded by the source corpus.

While many BERT-based cross-modal pre-trained models produce excellent results on downstream understanding tasks like image-text retrieval and VQA, they cannot be applied to generation tasks directly. In this paper, we propose XGPT, a new method of Cross-modal Generative Pre-Training for Image Captioning that is designed to pre-train text-to-image caption generators through three novel generation tasks, including Image-conditioned Masked Language Modeling (IMLM), Image-conditioned Denoising Autoencoding (IDA), and Text-conditioned Image Feature Generation (TIFG). As a result, the pre-trained XGPT can be fine-tuned without any task-specific architecture modifications to create state-of-the-art models for image captioning. Experiments show that XGPT obtains new state-of-the-art results on the benchmark datasets, including COCO Captions and Flickr30k Captions. We also use XGPT to generate new image captions as data augmentation for the image retrieval task and achieve significant improvement on all recall metrics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Multimodal pre-training has brought leaping advances in vision+language understanding tasks such as VQA and VCR, with great potential in extending to visual captioning [85, 71], visual dialog [47, 69], vision-language naviga-”
Pre-Trained Models: Past, Present and Future (2021, methodology): “X-GPT (Xia et al., 2020) finds that while previous BERT-based multimodal PTMs produce excellent results on downstream understanding tasks, they cannot be applied to generation tasks directly.”
DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2022, background): “…Chen et al., 2020; Dosovitskiy et al., 2020). pretrained models (PMs) such as pretrained language models (PLMs) (Devlin et al., 2018; Brown et al., 2020), and pretrained visual-language models (Lu et al., 2019; Li et al., 2019; Su et al., 2019; Xia et al., 2021) have advanced a lot in recent years.”
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, methodology): “Baseline methods include image captioning models BUTD (Anderson et al., 2018) and AoANet (Huang et al., 2019); vision-language generative models, e.g., VL-BART, VL-T5 (Cho et al., 2021), XGPT(Xia et al., 2021); models using additional auxiliary input such as UNITAB (Yang et al., 2022) with object de”
WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training (2021, background): “Most existing multi-modal pre-training models, especially those with the single-tower architecture [20, 35, 26, 38, 9, 11, 40, 22, 7, 14], take an assumption that there exists strong semantic correlation between the input imagetext pair.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “and entail predicting the original word sequence given a corrupted one [125].”
Kaleido-BERT: Vision-Language Pre-training on Fashion Domain (2021, unknown): “, image captioning like XGPT [73], image-text retrieval like Image-BERT [55]).”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “• When the input data x is multimedia input such as image and speech, it becomes image captioning [191] or speech recognition [45].”
VC-GPT: Visual Conditioned GPT for End-to-End Generative Vision-and-Language Pre-training (2022, methodology): “Notably, our model can achieve a CIDEr score of 127.7 on the MSCOCO test set using only 110k images for”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “Note that, the GPT-3 [12] has ten times more parameters than TuringNLP [60].”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, methodology): “For instance, XGPT [119] takes images as the input and uses the image captioning task as the primary task in the pretraining stage for GPT to generate images’ captions.”
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2020, unknown): “9 -/-/XGPT (w/ fine-tune on en) [35] 31.”