Compressing Visual-linguistic Model via Knowledge Distillation

By Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, Zicheng Liu. 2021

124 citations recorded by the source corpus.

Despite exciting progress in pre-training for visual-linguistic (VL) representations, very few aspire to a small VL model. In this paper, we study knowledge distillation (KD) to effectively compress a transformer based large VL model into a small VL model. The major challenge arises from the inconsistent regional visual tokens extracted from different detectors of Teacher and Student, resulting in the misalignment of hidden representations and attention distributions. To address the problem, we retrain and adapt the Teacher by using the same region proposals from Student’s detector while the features are from Teacher’s own object detector. With aligned network inputs, the adapted Teacher is capable of transferring the knowledge through the intermediate representations. Specifically, we use the mean square error loss to mimic the attention distribution inside the transformer block, and present a token-wise noise contrastive loss to align the hidden state by contrasting with negative representations stored in a sample queue. To this end, we show that our proposed distillation significantly improves the performance of small VL models on image captioning and visual question answering tasks. It reaches 120.8 in CIDEr score on COCO captioning, an improvement of 5.1 over its non-distilled counterpart; and an accuracy of 69.8 on VQA 2.0, a 0.8 gain from the baseline. Our extensive experiments and ablations confirm the effectiveness of VL distillation in both pre-training and fine-tuning stages.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhe Gan
3. Lijuan Wang
4. Zicheng Liu
5. Linjie Li
6. Jianwei Yang
7. Jianfeng Wang
8. Xiaowei Hu

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Zhe Gan
3. Luc Van Gool
4. Dacheng Tao
5. Lijuan Wang
6. Jianwei Yang
7. Zicheng Liu
8. Cordelia Schmid

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Jianfeng Gao
4. Linjie Li
5. Zicheng Liu
6. Jianfeng Wang
7. Xiaowei Hu
8. Jianwei Yang

In Efficiency & Compression

1. Zhe Gan
2. Yen-Chun Chen
3. Jianfeng Wang
4. Zicheng Liu
5. Lijuan Wang
6. Jingjing Liu
7. Xiaowei Hu
8. Yu Cheng

Citations in the corpus

• Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Along the journey of VLP, researchers have investigated different training strategies [11,31], robustness [25], compression [9, 10, 42], probing analysis [3, 27], and the extension to video-text modeling [21, 24, 38, 39, 50].”
• Florence: A New Foundation Model for Computer Vision (2021, methodology): “Thus, the object detector has been a de facto tool for image feature extraction, followed by a fusion network for prediction in many works (Anderson et al., 2018; Li et al., 2020; Zhang et al., 2021b; Wang et al., 2020; Fang et al., 2021; Chen et al., 2020d).”
• UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The image can be encoded as object features [5,6,13,18,27,29,30,30,32,39,42,47,53, 54] through a Faster RCNN [36], as grid features [20, 22] from a convolutional neural network, or as patch features [25] from a linear projection on raw pixels.”
• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “During pre-training, Masked Language Modeling (MLM) and Image-Text Matching (ITM) tasks have been widely used [106, 26, 66, 127, 13, 23, 105, 54].”
• Playing Lottery Tickets with Vision and Language (2021, background): “2020a) and DistilVLM (Fang et al. 2021), have also attempted to train a smaller VLP model; however, our focus is different from theirs.”
• Injecting Semantic Concepts into End-to-End Image Captioning (2021, methodology): “Previous efforts [18, 25, 38, 67, 79] have demonstrated that the object tags play an important role in improving the captioning performance.”
• VLP: A Survey on Vision-language Pre-training (2022, methodology): “In general PLMs, model compression and acceleration is a hot topic, and specific methods include parameter sharing[25], model pruning[145], knowledge distillation[146], and model quantization[147].”