MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning

By Constantin Eichenberg, Sid Black, Samuel Weinbach, Letitia Parcalabescu, Anette Frank. 2021

112 citations recorded by the source corpus.

Large-scale pretraining is fast becoming the norm in Vision-Language (VL) modeling. However, prevailing VL approaches are limited by the requirement for labeled data and the use of complex multi-step pretraining objectives. We present MAGMA - a simple method for augmenting generative language models with additional modalities using adapter-based finetuning. Building on Frozen, we train a series of VL models that autoregressively generate text from arbitrary combinations of visual and textual input. The pretraining is entirely end-to-end using a single language modeling objective, simplifying optimization compared to previous approaches. Importantly, the language model weights remain unchanged during training, allowing for transfer of encyclopedic knowledge and in-context learning abilities from language pretraining. MAGMA outperforms Frozen on open-ended generative tasks, achieving state of the art results on the OKVQA benchmark and competitive results on a range of other popular VL benchmarks, while pretraining on 0.2% of the number of samples used to train SimVLM.

Subfields: Multimodal Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Efficiency & Compression

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[26] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank.”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, background): “With the increased number of model parameters, it is hard to train those pretrained LLMs, thus more efforts have been paid to directly leverage the off-the-shelf frozen pre-trained LLMs for VL tasks [12, 38, 8, 46, 50].”
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, background): “Tsimpoukelli et al. (2021) train vision encoders that output soft prompts to frozen PLMs. Chen et al. (2022) and Eichenberg et al. (2021) train both the vision encoders and new layers inserted into PLMs.”
Mapping Language Models to Grounded Conceptual Spaces (2022, background): “In the case of vision, this could be to learn correspondences between language descriptions and pixels in an image (Eichenberg et al., 2021; Tsimpoukelli et al., 2021), or in the case of RL, to build agents that understand natural language instructions in order to take actions in a world that…”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, methodology): “Similar to MAGMA [22], we also use adapter layers [31, 32].”
MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting (2022, methodology): “Following previous works (Tsimpoukelli et al., 2021; Eichenberg et al., 2021), we train our model using a standard language modeling objective on image captions with teacher forcing (Lamb et al.”
Grounding Language Models to Images for Multimodal Generation (2023, methodology): “MAGMA (Eichenberg et al., 2022) improved upon Frozen by training adapter modules for improved performance on downstream tasks.”
Generating Images with Multimodal Language Models (2023, methodology): “We follow prior work [56, 19, 35, 31] in learning translation parameters that map from image features to text embedding space.”