Reveal: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory
By Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A. Ross, Alireza Fathi. 2022
178 citations recorded by the source corpus.
In this paper, we propose an end-to-end Retrieval-Augmented Visual Language Model (REVEAL) that learns to encode world knowledge into a large-scale memory, and to retrieve from it to answer knowledge-intensive queries. Reveal consists of four key components: the memory, the encoder, the retriever and the generator. The large-scale memory encodes various sources of multimodal world knowledge (e.g. image-text pairs, question answering pairs, knowledge graph triplets, etc.) via a unified encoder. The retriever finds the most relevant knowledge entries in the memory, and the generator fuses the retrieved knowledge with the input query to produce the output. A key novelty in our approach is that the memory, encoder, retriever and generator are all pre-trained end-to-end on a massive amount of data. Furthermore, our approach can use a diverse set of multimodal knowledge sources, which is shown to result in significant gains. We show that Reveal achieves state-of-the-art results on visual question answering and image captioning. The project page of this work is reveal. github. io.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Multimodal Learning, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Yu Qiao
- Luc Van Gool
- Dacheng Tao
- Kai Chen
- Trevor Darrell
- Andrew Zisserman
- Xiaogang Wang
- Jitendra Malik
In Multimodal Learning
In Representation & Self-Supervised Learning
- Yu Qiao
- Carl Vondrick
- D'idac Sur'is
- Sachit Menon
- Andrew Zisserman
- Tinne Tuytelaars
- Antonio Torralba
- David F. Harwath
Citations in the corpus
- ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “Due to the practical difficulty of using these methods, the field has primarily moved towards end-to-end all-in-one models [1, 22, 23, 30].”
- InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition (2023, methodology): “REVEAL [25] addresses this by introducing an end-to-end retrieval-enhanced visual language model that leverages various knowledge source modalities and operates in tandem with a generator, excelling in knowledge-intensive visual question-answering tasks.”
- Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “Till not, there lacks a sufficient comparison of different ways of interaction across various scenarios.”
- Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity (2026, unknown): “Multimodal RAG extensions remain limited: RA-Invoice CM3 [74] evaluates on-web images, REVEAL [27] uses Wikipedia images, and MuRAG [13] requires manual modality selection [51,74].”
- MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation (2026, unknown): “Multimodal RAG (MM-RAG) addresses this limitation by introducing external memory for grounding generation in retrieved visual–textual evidence [9, 32, 72].”