KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation

By Yongfei Liu, Chenfei Wu, Shao-Yen Tseng, Vasudev Lal, Xuming He, Nan Duan. 2021

33 citations recorded by the source corpus.

Self-supervised vision-and-language pretraining (VLP) aims to learn transferable multi-modal representations from large-scale image-text data and to achieve strong performances on a broad scope of vision-language tasks after finetuning. Previous mainstream VLP approaches typically adopt a two-step strategy relying on external object detectors to encode images in a multi-modal Transformer framework, which suffer from restrictive object concept space, limited image context and inefficient computation. In this paper, we propose an object-aware end-to-end VLP framework, which directly feeds image grid features from CNNs into the Transformer and learns the multi-modal representations jointly. More importantly, we propose to perform object knowledge distillation to facilitate learning cross-modal alignment at different semantic levels. To achieve that, we design two novel pretext tasks by taking object features and their semantic labels from external detectors as supervision: 1.) Object-guided masked vision modeling task focuses on enforcing object-aware representation learning in the multi-modal Transformer; 2.) Phrase-region alignment task aims to improve cross-modal alignment by utilizing the similarities between noun phrases and object labels in the linguistic space. Extensive experiments on a wide range of vision-language tasks demonstrate the efficacy of our proposed framework, and we achieve competitive or superior performances over the existing pretraining strategies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zicheng Liu
2. Lijuan Wang
3. Zhe Gan
4. Jianfeng Wang
5. Xiaowei Hu
6. Zhenguo Li
7. Lanqing Hong
8. Yezhou Yang

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Zicheng Liu
3. Luc Van Gool
4. Lijuan Wang
5. Dacheng Tao
6. Cordelia Schmid
7. Trevor Darrell
8. Jitendra Malik

Citations in the corpus

• Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “Single-stream models [10,24,31] introduce powerful encoders such as Transformer [44] to model the cross-modal interaction between image and text.”
• Injecting Semantic Concepts into End-to-End Image Captioning (2021, methodology): “Recent works [16,40] reveal that by leveraging the knowledge distillation technique [22], the VL model can be improved compared to the non-distilled counterpart using a pretrained Teacher VL model.”