Reveal: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory

By Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A. Ross, Alireza Fathi. 2022

178 citations recorded by the source corpus.

In this paper, we propose an end-to-end Retrieval-Augmented Visual Language Model (REVEAL) that learns to encode world knowledge into a large-scale memory, and to retrieve from it to answer knowledge-intensive queries. Reveal consists of four key components: the memory, the encoder, the retriever and the generator. The large-scale memory encodes various sources of multimodal world knowledge (e.g. image-text pairs, question answering pairs, knowledge graph triplets, etc.) via a unified encoder. The retriever finds the most relevant knowledge entries in the memory, and the generator fuses the retrieved knowledge with the input query to produce the output. A key novelty in our approach is that the memory, encoder, retriever and generator are all pre-trained end-to-end on a massive amount of data. Furthermore, our approach can use a diverse set of multimodal knowledge sources, which is shown to result in significant gains. We show that Reveal achieves state-of-the-art results on visual question answering and image captioning. The project page of this work is reveal. github. io.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yu Qiao
  2. Yuhang Cao
  3. D'idac Sur'is
  4. Kai Chen
  5. Songyang Zhang
  6. Carl Vondrick
  7. Xingcheng Zhang
  8. Conghui He

In Computer Vision and Pattern Recognition

  1. Yu Qiao
  2. Luc Van Gool
  3. Dacheng Tao
  4. Kai Chen
  5. Trevor Darrell
  6. Andrew Zisserman
  7. Xiaogang Wang
  8. Jitendra Malik

In Multimodal Learning

  1. Yu Qiao
  2. Kai Chen
  3. Carl Vondrick
  4. Conghui He
  5. Haodong Duan
  6. D'idac Sur'is
  7. Sachit Menon
  8. Kristen Grauman

In Representation & Self-Supervised Learning

  1. Yu Qiao
  2. Carl Vondrick
  3. D'idac Sur'is
  4. Sachit Menon
  5. Andrew Zisserman
  6. Tinne Tuytelaars
  7. Antonio Torralba
  8. David F. Harwath

Citations in the corpus