Retrieval-Augmented Multimodal Language Modeling
By Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Michael Lewis, Luke S. Zettlemoyer, Wen-tau Yih. 2023
166 citations recorded by the source corpus.
Recent multimodal models such as DALL-E and CM3 have achieved remarkable progress in textto-image and image-to-text generation. However, these models store all learned knowledge (e.g., the appearance of the Eiffel Tower) in the model parameters, requiring increasingly larger models and training data to capture more knowledge. To integrate knowledge in a more scalable and modular way, we propose a retrievalaugmented multimodal model, which enables a base multimodal model (generator) to refer to relevant knowledge fetched by a retriever from external memory (e.g., multimodal documents on the web). Specifically, we implement a retriever using the pretrained CLIP model and a generator using the CM3 Transformer architecture, and train this model using the LAION dataset. Our resulting model, named Retrieval-Augmented CM3 (RA-CM3), is the first multimodal model that can retrieve and generate mixtures of text and images. We show that RA-CM3 significantly outperforms baseline multimodal models such as DALL-E and CM3 on both image and caption generation tasks (12 FID and 17 CIDEr improvements on MSCOCO), while requiring much less compute for training (<30% of DALL-E). Moreover, we show that RA-CM3 exhibits novel capabilities such as knowledge-intensive image generation and multimodal in-context learning
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Large Language Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Noah A. Smith
- Graham Neubig
- Hannaneh Hajishirzi
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jianfeng Gao
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Cordelia Schmid
In Retrieval-Augmented Generation
- Danqi Chen
- Akari Asai
- Wen-tau Yih
- Luke S. Zettlemoyer
- Sewon Min
- Zexuan Zhong
- Michael Lewis
- Minjoon Seo
In Large Language Models
- Luke S. Zettlemoyer
- Jianfeng Gao
- Noah A. Smith
- Michael Lewis
- Danqi Chen
- Percy Liang
- Yulia Tsvetkov
- Wen-tau Yih
In Multimodal Learning
Citations in the corpus
- Scaling Laws for Generative Mixed-Modal Language Models (2023, background): “…language models have been developed for a wide range of data modalities, including nat-ural language text Brown et al. (2020), code (Chen et al., 2021; Fried et al., 2022), images (Ramesh et al., 2021; Yasunaga et al., 2022), and molecules or proteins (Chilingaryan et al., 2022; Hsu et al., 2022).”
- Retrieval-Augmented Generation for Large Language Models: A Survey (2023, background): “RA-CM3 [Yasunaga et al. , 2022] stands as a pio-neering multimodal model of both retrieving and generating text and images.”
- InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition (2023, background): “RA-CM3 [69] further enhances this process by retrieving external relevant knowledge as a supplement, facilitating superior generation of image and text results, and demonstrating remarkable performance in knowledge-intensive image generation and multimodal in-context learning tasks.”
- Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2023, methodology): “This setting is especially valuable to retrieval-augmented language models (Khandelwal et al., 2020; Shi et al., 2023; Min et al., 2022; Yasunaga et al., 2023), where documents retrieved from external databases are used as additional input to provide LMs additional knowledge.”
- REPLUG: Retrieval-Augmented Black-Box Language Models (2023, background): “Retrieval-augmented language models (Khandelwal et al., 2020; Borgeaud et al., 2022; Izacard et al., 2022b; Yasunaga et al., 2022), in contrast, can retrieve knowledge from an external datastore when needed, potentially reducing hallucination and increasing coverage.”
- Large Language Models as Analogical Reasoners (2023, background): “LLMs with billions of parameters demonstrate in-context learning and few-shot learning abilities (Brown et al., 2020; Liu et al., 2022; Su et al., 2022; Mishra et al., 2022; Wei et al., 2022a; Yasunaga et al., 2023; Shi et al., 2023).”
- Med-Flamingo: a Multimodal Medical Few-shot Learner (2023, background): “Recently, there has been a growing interest in extending language models to handle vision-language multimodal data and tasks (Su et al., 2019; Ramesh et al.; Alayrac et al., 2022; Aghajanyan et al.; Yasunaga et al., 2023).”
- Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, background): “Recent work has started to treat LMs as agents that can both reason and act [58, 33, 52, 36, 59].”
- Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, methodology): “Motivated by retrieval-augmented models in NLP, several recent works leverage visual and / or textual knowledge to improve classification [32], question answering [7, 35, 56, 60], image generation [2, 8, 49, 72], and multi-modal tasks simultaneously [62].”
- Retrieval-based Language Models and Applications (2023, methodology): “At this end, we will cover recent extensions of retrieval-based LMs beyond English text, including multilingual (Asai et al., 2021), multi-modal (Chen et al., 2022; Yasunaga et al., 2022) and code (Parvez et al., 2021) retrieval.”
- Generating Images with Multimodal Language Models (2023, methodology): “CM3 [2, 62] trained multimodal LMs on HTML webpages consisting of interleaved images and text.”
- Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity (2026, unknown): “Hybrid approaches like Flamingo [4] and RA-CM3 [74] combine modalities but focus on few-shot learning over natural image-text pairs rather than large-scale document retrieval.”
- MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation (2026, unknown): “Multimodal RAG (MM-RAG) addresses this limitation by introducing external memory for grounding generation in retrieved visual–textual evidence [9, 32, 72].”