Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions

By Liunian Harold Li, Haoxuan You, Zhecan Wang, Alireza Zareian, Shih-Fu Chang, Kai-Wei Chang. 2021

54 citations recorded by the source corpus.

Pre-trained contextual vision-and-language (V&L) models have achieved impressive performance on various benchmarks. However, existing models require a large amount of parallel image-caption data for pre-training. Such data are costly to collect and require cumbersome curation. Inspired by unsupervised machine translation, we investigate if a strong V&L representation model can be learned through unsupervised pre-training without image-caption corpora. In particular, we propose to conduct “mask-and-predict” pre-training on text-only and image-only corpora and introduce the object tags detected by an object recognition model as anchor points to bridge two modalities. We find that such a simple approach achieves performance close to a model pre-trained with aligned data, on four English V&L benchmarks. Our work challenges the widely held notion that aligned data is necessary for V&L pre-training, while significantly reducing the amount of supervision needed for V&L models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Lijuan Wang
3. Lu Yuan
4. Zhe Gan
5. Pengchuan Zhang
6. Xiaowei Hu
7. Xiyang Dai
8. Jianwei Yang

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Luc Van Gool
3. Lu Yuan
4. Lijuan Wang
5. Dacheng Tao
6. Trevor Darrell
7. Yi Yang
8. Cordelia Schmid

In Multimodal Learning

1. Lijuan Wang
2. Jianfeng Gao
3. Xiaowei Hu
4. Pengchuan Zhang
5. Lu Yuan
6. Zhe Gan
7. Xiyang Dai
8. Yen-Chun Chen

In Representation & Self-Supervised Learning

1. Jianfeng Gao
2. Lu Yuan
3. Lijuan Wang
4. Pengchuan Zhang
5. Xiyang Dai
6. Xiao Wang
7. Hamed Pirsiavash
8. Jenq-Neng Hwang

Citations in the corpus

• GLIPv2: Unifying Localization and Vision-Language Understanding (2022, methodology): “, BUTD) [2, 67] rely on a pre-trained localization model as their visual encoder; the downside is the pro-longed “localization->VLP” pre-training pipeline [45, 55, 13, 54, 38, 36, 71, 41, 39].”
• A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision (2021, background): “U-VisualBERT [28] explores removing paired vision and language data during pre-training.”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “60 U-VisualBERT [173] NAACL-2021 image-text Trans, BERT GR, MML Unpaired image-text data for pre-training URL”