MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting

By Oscar Mañas, Pau Rodr'iguez L'opez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, Aishwarya Agrawal. 2022

59 citations recorded by the source corpus.

Large pre-trained models have proved to be remarkable zero- and (prompt-based) few-shot learners in unimodal vision and language tasks. We propose MAPL, a simple and parameter-efficient method that reuses frozen pre-trained unimodal models and leverages their strong generalization capabilities in multimodal vision-language (VL) settings. MAPL learns a lightweight mapping between the representation spaces of unimodal models using aligned image-text data, and can generalize to unseen VL tasks from just a few in-context examples. The small number of trainable parameters makes MAPL effective at low-data and in-domain learning. Moreover, MAPL’s modularity enables easy extension to other pre-trained models. Extensive experiments on several visual question answering and image captioning benchmarks show that MAPL achieves superior or competitive performance compared to similar methods while training orders of magnitude fewer parameters. MAPL can be trained in just a few hours using modest computational resources and public datasets. We release our code and pre-trained model weights at https://github.com/oscmansan/mapl.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Rita Singh
2. Huaming Wang
3. Benjamin Elizalde
4. Soham Deshmukh

In Computer Vision and Pattern Recognition

1. Rita Singh
2. Huaming Wang
3. Benjamin Elizalde
4. Soham Deshmukh

In Multimodal Learning

1. Rita Singh
2. Huaming Wang
3. Benjamin Elizalde
4. Soham Deshmukh

In Efficiency & Compression

1. Rita Singh
2. Huaming Wang
3. Benjamin Elizalde
4. Soham Deshmukh

Citations in the corpus

• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “Some methods freeze the language model to use the knowledge from LLMs for vision-to-language generation tasks (Tsimpoukelli et al., 2021; Alayrac et al., 2022; Chen et al., 2022a; Ma˜nas et al., 2023; Tiong et al., 2022; Guo et al., 2022).”
• Pengi: An Audio Language Model for Audio Tasks (2023, methodology): “Models in Computer Vision [44, 2, 41] use a frozen image encoder like CLIP, but CLAP is trained on a magnitude smaller collection of audio-text pairs.”
• What matters when building vision-language models? (2024, unknown): “We compare multiple pooling strategies: a simple linear layer that takes the flattened sequence of vision hidden states and projects it into a shorter sequence of visual tokens, as well as a Mapping Network (Mañas et al., 2023).”