Distilling Knowledge from Reader to Retriever for Question Answering

By Gautier Izacard, Edouard Grave. 2020

323 citations recorded by the source corpus.

The task of information retrieval is an important component of many natural language processing systems, such as open domain question answering. While traditional methods were based on hand-crafted features, continuous representations based on neural networks recently obtained competitive results. A challenge of using such methods is to obtain supervised data to train the retriever model, corresponding to pairs of query and support documents. In this paper, we propose a technique to learn retriever models for downstream tasks, inspired by knowledge distillation, and which does not require annotated pairs of query and documents. Our approach leverages attention scores of a reader model, used to solve the task based on retrieved documents, to obtain synthetic labels for the retriever. We evaluate our method on question answering, obtaining state-of-the-art results.

Subfields: Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “On the ﬁltered version, Chinchilla lags behind the open book SOTA (Izacard and Grave, 2020) by only 7.9%.”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “In particular, Izacard & Grave (2021) showed that these scores can be aggregated across attention heads, layers and tokens for a given document to obtain a single score for each document.”
Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “…(Yarowsky, 1995), relation extraction (Brin, 1999; Agichtein and Gravano, 2000), parsing (McClosky et al., 2006), sequence generation (He et al., 2020), few-shot text classification (Schick and Schütze, 2021a) and retrieval (Izacard and Grave, 2021) to reasoning (Zelikman et al., 2022).”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, methodology): “, 2021), but does not use a modular approach to generate knowledge, instead applying the fusion-indecoder (FiD) method (Izacard and Grave, 2021a) to output a response directly given the retrieved documents.”
You Only Need One Model for Open-domain Question Answering (2021, methodology): “In contrast, our method uses only 83B tokens, which is 76.4% and 51.5% less than the training tokens used to train REALM and EMDR 2 respectively.”
Hindsight: Posterior-guided training of retrievers for improved open-ended generation (2021, methodology): “Comparison with Fusion-in-Decoder Izacard & Grave (2021a) provide multiple passages to the generator simultaneously and in Izacard & Grave (2021b) they use the decoder’s attention weights over each passage for relevance supervision.”
R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, background): “ing large quantities of retrieved passages Izacard and Grave (2021). They compensate for a certain amount of the retrieval error and enable early aggregation of answer’s evidence between passages.”
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, methodology): “First, they adopt a stage-wise training, where the retriever is trained while freezing the reader and vice versa (Karpukhin et al., 2020, Izacard and Grave, 2021b,a).”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, background): “, 2020), distill knowledge from the retriever to the reader (Izacard and Grave, 2021), and boost performance in multi-hop or complex QA settings (Xiong et al.”
A Memory Efficient Baseline for Open Domain Question Answering (2020, methodology): “Several works showed that generative QA models could be improved by retrieval, using the obtained documents as input of the sequenceto-sequence model (Min et al., 2020; Lewis et al., 2020; Izacard and Grave, 2020b). ar X iv :2 01 2.”
Recursively Summarizing Books with Human Feedback (2021, methodology): “However, we use far more parameters than Izacard and Grave (2020), the previous SOTA.”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, neutral): “It’s interesting to compare this to more direct distillation methods (Izacard and Grave, 2020; Yang and Seo, 2020), which also re-ported similar gains.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “training 2020 AugmentSBERT [102] BERTbase SR NLL SHN+DeHN KD Kernel density estimation 2020 TCT-ColBERT [103] BERTbase SR KL IB+SHN KD Sparse-dense hybrid 2020 Multi-stage [104] BERTbase SR NLL IB+SHN KD TAP Embedding fusion 2020 DKRR [105] BERTbase SR KL SHN KD Iter.”
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021, background): “, 2021) and (2) distilling the knowledge from cross-encoder-based reranker into dual-encoder-based retriever (Izacard and Grave, 2020; Yang and Seo, 2020; Qu et al., 2021; Ren et al., 2021).”
Boosted Dense Retriever (2021, background): “Distillation for dense retrievers is an active area, and more complex schemes exist which could improve results further (Izacard and Grave, 2021; Qu et al., 2021; Yang and Seo, 2020; Lin et al., 2021; Hofst¨atter et al., 2021a; Barkan et al., 2020; Gao et al., 2020).”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, unknown): “…similar to the retrieve-and-rerank approach is common— research aims to synchronize the models through knowledge distillation from the reader to the re-triever (Yang and Seo, 2020; Izacard and Grave, 2021) or by directly training both models end-to-end (Lee et al., 2019; Sachan et al., 2021a,b).”
PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, unknown): “Prior works [13, 36, 63, 72, 40, 41, 17, 18, 12] investigate leveraging knowledge”
Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering (2022, methodology): “We use the averaged cross-attention score of the generative model to select supporting facts [18].”
Reader-Guided Passage Reranking for Open-Domain Question Answering (2021, background): “Recent studies (Izacard and Grave, 2020a; Yang and Seo, 2020) show that distillation from the preference of the reader can improve the retriever performance, where the reader preference is measured by the attention scores of the reader over different passages and the retriever is reﬁned by learning…”