Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
By Ming Yan, Haiyang Xu, Chenliang Li, Bin Bi, Junfeng Tian, Min Gui, Wei Wang. 2021
11 citations recorded by the source corpus.
Existing approaches to vision-language pre-training (VLP) heavily rely on an object detector based on bounding boxes (regions), where salient objects are first detected from images and then a Transformer-based model is used for cross-modal fusion. Despite their superior performance, these approaches are bounded by the capability of the object detector in terms of both effectiveness and efficiency. Besides, the presence of object detection imposes unnecessary constraints on model designs and makes it difficult to support end-to-end training. In this paper, we revisit grid-based convolutional features for vision-language pre-training, skipping the expensive region-related steps. We propose a simple yet effective grid-based VLP method that works surprisingly well with the grid features. By pre-training only with in-domain datasets, the proposed Grid-VLP method can outperform most competitive region-based VLP methods on three examined vision-language understanding tasks. We hope that our findings help to further advance the state of the art of vision-language pre-training, and provide a new direction towards effective and efficient VLP.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Zicheng Liu
- Zhe Gan
- Lijuan Wang
- Jianfeng Wang
- Rita Cucchiara
- Xiaowei Hu
- Lorenzo Baraldi
- Marcella Cornia
In Multimodal Learning
- Lijuan Wang
- Jianfeng Wang
- Zicheng Liu
- Zhe Gan
- Xiaowei Hu
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
In Representation & Self-Supervised Learning
- Lijuan Wang
- Zicheng Liu
- Zhe Gan
- Rita Cucchiara
- Lorenzo Baraldi
- Zhiyuan Fang
- Jianfeng Wang
- Xiaowei Hu
Citations in the corpus
- Universal Captioner: Long-Tail Vision-and-Language Model Training through Content-Style Separation (2021, methodology): “Like other works that employ encoder-decoder structures [60, 62], SimVLM [59] neglects the usage of BERT-like structures and learns visual features from scratch, training on the large, not publicly available, ALIGN [23].”
- Injecting Semantic Concepts into End-to-End Image Captioning (2021, background): “To address these challenges, there is an emerging trend that more recent works propose to eliminate the detector for the VL pre-training in an end-to-end fashion [25, 26, 30, 61, 64].”