Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

By Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, Pascale Fung. 2022

113 citations recorded by the source corpus.

The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with 7\times fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “With the vast advancement of the Transformer architecture [35, 189] in both CV and NLP, there is a trend to pre-train large-scale unified vision-language (VL) models [1, 27, 105, 196, 197, 200] to perform vision grounded text generation tasks, such as image captioning and visual question answering (”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
Language Models are General-Purpose Interfaces (2022, background): “, 2021) and VLKD (Dai et al., 2022), even though Frozen has significantly more parameters.”
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, methodology): “VLKD (Dai et al., 2022) distills multimodal knowledge to PLM by using CLIP (Radford et al.”
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, background): “…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting (2022, methodology): “Next, MAPL CC-clean is competitive compared to existing methods (MAGMA, VLKD, ClipCap) and concurrent work LiMBeR, despite training one-two orders of magnitude fewer parameters on significantly less multimodal data.”
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2022, methodology): “, 2020), text infilling (Dai et al., 2022; Wang et al., 2022), prefix LM (Wang et al.”