Is Retriever Merely an Approximator of Reader?

By Sohee Yang, Minjoon Seo. 2020

44 citations recorded by the source corpus.

The state of the art in open-domain question answering (QA) relies on an efficient retriever that drastically reduces the search space for the expensive reader. A rather overlooked question in the community is the relationship between the retriever and the reader, and in particular, if the whole purpose of the retriever is just a fast approximation for the reader. Our empirical evidence indicates that the answer is no, and that the reader and the retriever are complementary to each other even in terms of accuracy only. We make a careful conjecture that the architectural constraint of the retriever, which has been originally intended for enabling approximate search, seems to also make the model more robust in large-scale search. We then propose to distill the reader into the retriever so that the retriever absorbs the strength of the reader while keeping its own benefit. Experimental results show that our method can enhance the document recall rate as well as the end-to-end QA accuracy of off-the-shelf retrievers in open-domain QA tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

In Information Retrieval & Search

Citations in the corpus

Distilling Knowledge from Reader to Retriever for Question Answering (2020, background): “Simultaneously to our work, Yang & Seo (2020) proposes to train a retriever with knowledge distillation.”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “In the context of question answering, knowledge distillation has been used to train retrievers, either using the attention scores of the reader of the downstream task as synthetic labels (Izacard & Grave, 2020a), or the relevance score from a cross encoder (Yang & Seo, 2020).”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, background): “It’s interesting to compare this to more direct distillation methods (Izacard and Grave, 2020; Yang and Seo, 2020), which also re-ported similar gains.”
Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “In the context of question answering, knowledge distillation has been used to train retrievers, either using the attention scores of the reader of the downstream task as synthetic labels (Izacard & Grave, 2021), or the relevance score from a cross encoder (Yang & Seo, 2020).”
Reader-Guided Passage Reranking for Open-Domain Question Answering (2021, background): “Recent studies (Izacard and Grave, 2020a; Yang and Seo, 2020) show that distillation from the preference of the reader can improve the retriever performance, where the reader preference is measured by the attention scores of the reader over different passages and the retriever is reﬁned by learning…”
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021, background): “…(1) incorporating hard negatives during training (Karpukhin et al., 2020; Xiong et al., 2020a; Qu et al., 2021) and (2) dis-tilling the knowledge from cross-encoder-based reranker into dual-encoder-based retriever (Izac-ard and Grave, 2020; Yang and Seo, 2020; Qu et al., 2021; Ren et al., 2021).”
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer (2022, methodology): “Inspired by reader-to-retriever distillation (Izacard and Grave, 2021a; Yang and Seo, 2020), we measure document usefulness based on cross-attention between decoder and encoder, and minimize retrieval atten-tion’s discrepancy from it through distillation.”
Phrase Retrieval Learns Passage Retrieval, Too (2021, background): “† : (Yang and Seo, 2020).”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, unknown): “…similar to the retrieve-and-rerank approach is common— research aims to synchronize the models through knowledge distillation from the reader to the re-triever (Yang and Seo, 2020; Izacard and Grave, 2021) or by directly training both models end-to-end (Lee et al., 2019; Sachan et al., 2021a,b).”
Boosted Dense Retriever (2021, background): “Distillation for dense retrievers is an active area, and more complex schemes exist which could improve results further (Izacard and Grave, 2021; Qu et al., 2021; Yang and Seo, 2020; Lin et al., 2021; Hofst¨atter et al., 2021a; Barkan et al., 2020; Gao et al., 2020).”