Scaling Up Vision-Language Pretraining for Image Captioning

By Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, Lijuan Wang. 2021

335 citations recorded by the source corpus.

In recent years, we have witnessed significant performance boost in the image captioning task based on vision-language pre-training (VLP). Scale is believed to be an important factor for this advance. However, most existing work only focuses on pre-training transformers with moderate sizes (e.g., 12 or 24 layers) on roughly 4 million images. In this paper, we present LEMON O, a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning. We use the state-of-the-art Vin VL model as our reference model, which consists of an image feature extractor and a transformer model, and scale the transformer both up and down, with model sizes ranging from 13 to 675 million parameters. In terms of data, we conduct experiments with up to 200 million imagetext pairs which are automatically collected from web based on the alt attribute of the image (dubbed as ALT200M11The dataset is released at https://github.com/xiaoweihu/ALT200M). Extensive analysis helps to characterize the performance trend as the model size and the pre-training data size increase. We also compare different training recipes, especially for training on large-scale noisy data. As a result, LEMON achieves new state of the arts on several major image captioning benchmarks, including COCO Caption, nocaps, and Conceptual Captions. We also show LEMON can generate captions with long-tail vi-sual concepts when used in a zero-shot manner.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, neutral): “Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
PaLM-E: An Embodied Multimodal Language Model (2023, background): “, 2021b), captioning (Hu et al., 2022), optical character recognition (Li et al.”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “Building datasets from alt-text continued with ALT200M [26] and ALIGN [28], which increased the dataset size up to 1.8 billion image-text pairs.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “The reason could be that LEMON uses MLM, instead of LM, which might be more difficult to train.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “With more relaxed filtering, LEMON (Hu et al., 2022) collected a larger dataset with 200M examples, which is further expanded to 800M examples in GIT (Wang et al.”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “[34] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang.”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “Pretraining large-scale video-text models and ﬁne-tuning them for down-stream tasks has become the norm in the video-language ﬁeld [21–23, 19, 24, 23, 15, 25–31, 4, 5, 32– 34].”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, background): “Vision language models (VLMs) have made significant progress in recent years across a variety of tasks including image cap-tioning, visual question answering (VQA), embodied planning, action recognition, and more [2, 18, 24, 31].”
Multimodal Learning With Transformers: A Survey (2022, methodology): “In general, existing multimodal Transformer models [9], [199], [263] are superior only for specific MML tasks, as they are designed specifically for only a subset of specific tasks [137], [142], [212], [249], [260], [261], [265], [266].”
Reproducible Scaling Laws for Contrastive Language-Image Learning (2022, background): “Earlier efforts to provide open image-text datasets like MS-COCO [46], Visual Genome [42], YFCC-100M [74], Conceptual Captions CC3M and CC12M [67, 30] do not match the current scale of private data used to train large-scale language vision models.”
Translation between Molecules and Natural Language (2022, background): “Recent state-of-the-art models tend to pretrain transformer-based models on massive textimage corpora (Li et al., 2020b; Hu et al., 2021).”
Sigmoid Loss for Language Image Pre-Training (2023, methodology): “Seminal works CLIP [34] and ALIGN [23] established the viability of this approach at a large scale, and following their success, many large image-text datasets became available privately [55, 13, 21, 45] and publicly [37, 6, 15, 7, 38].”