MiniVLM: A Smaller and Faster Vision-Language Model

By Jianfeng Wang, Xiaowei Hu, Pengchuan Zhang, Xiujun Li, Lijuan Wang, L. Zhang, Jianfeng Gao, Zicheng Liu. 2020

76 citations recorded by the source corpus.

Recent vision-language (VL) studies have shown remarkable progress by learning generic representations from massive image-text pairs with transformer models and then fine-tuning on downstream VL tasks. While existing research has been focused on achieving high accuracy with large pre-trained models, building a lightweight model is of great value in practice but is less explored. In this paper, we propose a smaller and faster VL model, MiniVLM, which can be finetuned with good performance on various downstream tasks like its larger counterpart. MiniVLM consists of two modules, a vision feature extractor and a transformer-based vision-language fusion module. We design a Two-stage Efficient feature Extractor (TEE), inspired by the one-stage EfficientDet network, to significantly reduce the time cost of visual feature extraction by $95\%$, compared to a baseline model. We adopt the MiniLM structure to reduce the computation cost of the transformer module after comparing different compact BERT models. In addition, we improve the MiniVLM pre-training by adding $7M$ Open Images data, which are pseudo-labeled by a state-of-the-art captioning model. We also pre-train with high-quality image tags obtained from a strong tagging model to enhance cross-modality alignment. The large models are used offline without adding any overhead in fine-tuning and inference. With the above design choices, our MiniVLM reduces the model size by $73\%$ and the inference time cost by $94\%$ while being able to retain $94-97\%$ of the accuracy on multiple VL tasks. We hope that MiniVLM helps ease the use of the state-of-the-art VL research for on-the-edge applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhe Gan
3. Lijuan Wang
4. Zicheng Liu
5. Linjie Li
6. Jianwei Yang
7. Jianfeng Wang
8. Chunyuan Li

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Zhe Gan
3. Luc Van Gool
4. Dacheng Tao
5. Cordelia Schmid
6. Lijuan Wang
7. Zicheng Liu
8. Jianwei Yang

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Jianfeng Gao
4. Linjie Li
5. Zicheng Liu
6. Jianfeng Wang
7. Xiaowei Hu
8. Zhengyuan Yang

In Efficiency & Compression

1. Zhe Gan
2. Yen-Chun Chen
3. Jianfeng Wang
4. Zicheng Liu
5. Lijuan Wang
6. Jingjing Liu
7. Xiaowei Hu
8. Yu Cheng

Citations in the corpus

• Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Along the journey of VLP, researchers have investigated different training strategies [11,31], robustness [25], compression [9, 10, 42], probing analysis [3, 27], and the extension to video-text modeling [21, 24, 38, 39, 50].”
• Florence: A New Foundation Model for Computer Vision (2021, methodology): “Thus, the object detector has been a de facto tool for image feature extraction, followed by a fusion network for prediction in many works (Anderson et al., 2018; Li et al., 2020; Zhang et al., 2021b; Wang et al., 2020; Fang et al., 2021; Chen et al., 2020d).”
• UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The image can be encoded as object features [5,6,13,18,27,29,30,30,32,39,42,47,53, 54] through a Faster RCNN [36], as grid features [20, 22] from a convolutional neural network, or as patch features [25] from a linear projection on raw pixels.”
• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “During pre-training, Masked Language Modeling (MLM) and Image-Text Matching (ITM) tasks have been widely used [106, 26, 66, 127, 13, 23, 105, 54].”
• Compressing Visual-linguistic Model via Knowledge Distillation (2021, methodology): “Following MiniVLM [71], we also adopt the EfficientNet [66] based lightweight object detector (TEE) for visual feature extraction.”
• Playing Lottery Tickets with Vision and Language (2021, background): “Note that a recent work, MiniVLM [69], has also attempted to train a smaller V+L model; however, our focus is quite different from theirs.”
• PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, background): “Following the success of pretrained language models [6, 13, 14, 30, 48] in the field of NLP, numerous works for vision and language (VL) tasks [25, 32, 65, 76, 82] have achieved great success.”