MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

By Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, William W. Cohen. Conference on Empirical Methods in Natural Language Processing, 2022

304 citations recorded by the source corpus.

While language Models store a massive amount of world knowledge implicitly in their parameters, even very large models often fail to encode information about rare entities and events, while incurring huge computational costs. Recently, retrieval-augmented models, such as REALM, RAG, and RETRO, have incorporated world knowledge into language generation by leveraging an external non-parametric index and have demonstrated impressive performance with constrained model sizes. However, these methods are restricted to retrieving only textual knowledge, neglecting the ubiquitous amount of knowledge in other modalities like images – much of which contains information not covered by any text. To address this limitation, we propose the first Multimodal Retrieval-Augmented Transformer (MuRAG), which accesses an external non-parametric multimodal memory to augment language generation. MuRAG is pre-trained with a mixture of large-scale image-text and text-only corpora using a joint contrastive and generative loss. We perform experiments on two different datasets that require retrieving and reasoning over both images and text to answer a given query: WebQA, and MultimodalQA. Our results show that MuRAG achieves state-of-the-art accuracy, outperforming existing models by 10-20% absolute on both datasets and under both distractor and full-wiki settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/38b0803b59e4973f09018ce942164b02be4b8bc9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Retrieval-Augmented Multimodal Language Modeling (2023, background): “Diffusion ✔ Re-Imagen (Chenetal.)”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, methodology): “Motivated by retrieval-augmented models in NLP, several recent works leverage visual and / or textual knowledge to improve classification [32], question answering [7, 35, 56, 60], image generation [2, 8, 49, 72], and multi-modal tasks simultaneously [62].”
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, unknown): “…are required in practical applications, such as Multi-Image Reasoning (Suhr et al., 2018; Lu et al., 2021; Jiang et al., 2024), Multimodal RAG (Chen et al., 2022; Lin et al., 2024), Video Understanding (Xiao et al., 2021; Li et al., 2023c; Fu et al., 2024a; Wu et al., 2024), Multi-modal…”
Retrieval-based Language Models and Applications (2023, methodology): “At this end, we will cover recent extensions of retrieval-based LMs beyond English text, including multilingual (Asai et al., 2021), multi-modal (Chen et al., 2022; Yasunaga et al., 2022) and code (Parvez et al., 2021) retrieval.”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Most works [10, 32,37,39,56,68] learn large pre-trained models from massive vision-language data and ﬁnetune them for downstream tasks, which are usually extremely computationally expensive and time-consuming.”
Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agents (2026, unknown): “[14] Architecture : Flat multimodal store ( LinearStorage ).”
Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity (2026, unknown): “Multimodal RAG extensions remain limited: RA-Invoice CM3 [74] evaluates on-web images, REVEAL [27] uses Wikipedia images, and MuRAG [13] requires manual modality selection [51,74].”
MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation (2026, unknown): “Multimodal RAG (MM-RAG) addresses this limitation by introducing external memory for grounding generation in retrieved visual–textual evidence [9, 32, 72].”
Fix Before Search: Benchmarking Agentic Query Visual Pre-processing in Multimodal Retrieval-augmented Generation (2026, unknown): “Key hyperparameters include: LoRA rank r = 8 applied to all linear layers ( lora target=all ), learning rate 1 × 10 − 4 with cosine decay, and 3 training epochs over 1,000 refinement triplets from our benchmark.”
MiRAGE: A Multiagent Framework for Generating Multimodal Multihop Question-Answer Dataset for RAG Evaluation (2026, unknown): “MuRAG (Chen et al., 2022) requires joint reasoning over images and text.”
Towards Text-Image Interleaved Retrieval (2025, unknown): “…retrieval (MIR) aims to retrieve relevant information involving multiple modalities (Wei et al., 2024), which plays a crucial role in various applications such as e-commerce search (Wu et al., 2021) and retrieval augmented generation (RAG) systems (Chen et al., 2022; Ya-sunaga et al., 2023).”
Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (2024, unknown): “In vision-language RAG, models like MuRAG [9] leverage image-text memory to retrieve top-k neighbors by comparing inner-product similarities.”