Retrieval-Augmented Multimodal Language Modeling

By Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Michael Lewis, Luke S. Zettlemoyer, Wen-tau Yih. 2023

166 citations recorded by the source corpus.

Recent multimodal models such as DALL-E and CM3 have achieved remarkable progress in textto-image and image-to-text generation. However, these models store all learned knowledge (e.g., the appearance of the Eiffel Tower) in the model parameters, requiring increasingly larger models and training data to capture more knowledge. To integrate knowledge in a more scalable and modular way, we propose a retrievalaugmented multimodal model, which enables a base multimodal model (generator) to refer to relevant knowledge fetched by a retriever from external memory (e.g., multimodal documents on the web). Specifically, we implement a retriever using the pretrained CLIP model and a generator using the CM3 Transformer architecture, and train this model using the LAION dataset. Our resulting model, named Retrieval-Augmented CM3 (RA-CM3), is the first multimodal model that can retrieve and generate mixtures of text and images. We show that RA-CM3 significantly outperforms baseline multimodal models such as DALL-E and CM3 on both image and caption generation tasks (12 FID and 17 CIDEr improvements on MSCOCO), while requiring much less compute for training (<30% of DALL-E). Moreover, we show that RA-CM3 exhibits novel capabilities such as knowledge-intensive image generation and multimodal in-context learning

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Luke S. Zettlemoyer
  2. Jianfeng Gao
  3. Mohit Bansal
  4. Percy Liang
  5. Noah A. Smith
  6. Graham Neubig
  7. Hannaneh Hajishirzi
  8. Quoc V. Le

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Jianfeng Gao
  5. Xiaogang Wang
  6. Ming-Hsuan Yang
  7. Jitendra Malik
  8. Cordelia Schmid

In Retrieval-Augmented Generation

  1. Danqi Chen
  2. Akari Asai
  3. Wen-tau Yih
  4. Luke S. Zettlemoyer
  5. Sewon Min
  6. Zexuan Zhong
  7. Michael Lewis
  8. Minjoon Seo

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Jianfeng Gao
  3. Noah A. Smith
  4. Michael Lewis
  5. Danqi Chen
  6. Percy Liang
  7. Yulia Tsvetkov
  8. Wen-tau Yih

In Multimodal Learning

  1. Chunyuan Li
  2. Jianfeng Gao
  3. Jianwei Yang
  4. Yong Jae Lee
  5. Haotian Liu
  6. Yu Qiao
  7. Percy Liang
  8. Conghui He

Citations in the corpus