ReQA: An Evaluation for End-to-End Answer Retrieval Models

By Amin Ahmad, Noah Constant, Yinfei Yang, Daniel Matthew Cer. 2019

56 citations recorded by the source corpus.

Popular QA benchmarks like SQuAD have driven progress on the task of identifying answer spans within a specific passage, with models now surpassing human performance. However, retrieving relevant answers from a huge corpus of documents is still a challenging problem, and places different requirements on the model architecture. There is growing interest in developing scalable answer retrieval models trained end-to-end, bypassing the typical document retrieval step. In this paper, we introduce Retrieval Question-Answering (ReQA), a benchmark for evaluating large-scale sentence-level answer retrieval models. We establish baselines using both neural encoding models as well as classical information retrieval techniques. We release our evaluation code to encourage further work on this challenging task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, methodology): “retrieval, recent approaches in dense retrieval ﬁrst learn a distributed representation space of the query and documents, in which the relevance function f(q;d) can be a simple similarity calculation [9, 10, 11, 12, 13, 15, 23]. 2Code, trained models, and pre-computed embeddings are available at (ht”
Pre-training Tasks for Embedding-based Large-scale Retrieval (2020, methodology): “Following Ahmad et al. (2019), we split a passage into sentences, p = s 1 s 2 . . . s n and transform the original entry ( q , a , p ) to a new tuple ( q , s i , p ) where s i is the sentence contains the answer span a .”
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2021, methodology): “To our knowledge, we are the ﬁrst to study using large-scale pre-trained text-to-text models for sentence representation learning and to scale sentence embedding models up to 11 billion parameters.”
Open-Retrieval Conversational Question Answering (2020, methodology): “Specifically, our retriever adopts a dual-encoder architecture [1, 11, 21, 26] that uses separate ALBERT [24] encoders for questions and passages.”
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval (2020, methodology): “Similar observations are also found in prior research (Ahmad et al., 2019).”
Conformer-Kernel with Query Term Independence for Document Retrieval (2020, methodology): “…other, and efficient retrieval is achieved using approximate nearestneighbour search [Lee et al., 2019, Chang et al., 2020, Karpukhin et al., 2020, Ahmad et al., 2019, Khattab and Zaharia, 2020] or by employing other data structures, such as learning an inverted index based on latent…”
Zero-shot Neural Retrieval via Domain-targeted Synthetic Query Generation (2020, methodology): “We convert the original format to a retrieval task, where the goal is to retrieval the long answer among all wiki paragraphs (Ahmad et al., 2019 use to evaluate our models.”