Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering

By Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, G. Tesauro, Murray Campbell. 2017

166 citations recorded by the source corpus.

A popular recent approach to answering open-domain questions is to first search for question-related passages and then apply reading comprehension models to extract answers. Existing methods usually extract answers from single passages independently. But some questions require a combination of evidence from across different sources to answer correctly. In this paper, we propose two models which make use of multiple passages to generate their answers. Both use an answer-reranking approach which reorders the answer candidates generated by an existing state-of-the-art QA model. We propose two methods, namely, strength-based re-ranking and coverage-based re-ranking, to make use of the aggregated evidence from different passages to better determine the answer. Our models have achieved state-of-the-art results on three public open-domain QA datasets: Quasar-T, SearchQA and the open-domain version of TriviaQA, with about 8 percentage points of improvement over the former two datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “Self-attention has been used in several prior works (Cheng et al., 2016; Wang et al., 2017b; Pan et al., 2017).”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “Some work optimizes the retrieval module to aid in a speciﬁc, downstream task such as question answering, using search [40], reinforcement learning [6, 57, 56], or a latent variable approach [27, 18] as in our work.”
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “Lee et al. (2018) introduced a supervised learning method to rerank paragraphs based on BiLSTM, while Wang et al. (2018a) trained a ranking system with reinforcement learning.”
How Context Affects Language Models' Factual Predictions (2020, background): “…2017, Wang et al., 2018b,c, Lin et al., 2018, Min et al., 2018, Lee et al., 2018, 2019, Das et al., 2019, Xiong et al., 2019], improving answer aggregation [Clark and Gardner, 2018, Wang et al., 2018c, Lee et al., 2018, Pang et al., 2019], and accelerating the whole pipeline [Seo et al., 2019].”
Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, background): “Wang et al. (2018a) concurrently introduced a neural model that extracts evidence specifically for the correct answer, as an intermediate step in a QA pipeline.”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, background): “…this formulation, a couple of recent studies have proposed to improve the system us-ing stronger reading comprehension models (Yang et al., 2019; Wang et al., 2018), more effective learning objectives (Clark and Gardner, 2018; Min et al., 2019a; Wang et al., 2019) or paragraph reranking models…”
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, methodology): “Although using a single dump for different open-domain QA datasets is a common practice (Chen et al., 2017; Wang et al., 2018a; Lee et al., 2018), this potentially causes inconsistent or even unfair evaluation across different experimental settings, due to the temporal inconsistency of the…”
Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering (2019, methodology): “To answer these questions, we conduct a series of empirical studies on the OpenSQuAD dataset (Rajpurkar et al., 2016; Wang et al., 2018a 4) explicit inter-sentence matching is not helpful for BERT.”
Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index (2019, background): “Several studies have also shown that answer aggregation modules could improve performance of the pipelined models (Wang et al., 2018b; Lee et al., 2018).”
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering (2019, methodology): “Open-domain QA systems such as R 3 (Wang et al., 2018a) and DS-QA (Lin et al., 2018) have sophisticated retriever models where the reader and retriever are jointly trained.”
Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering (2019, background): “With just fine tuning on the source dataset, we obtain a score that is only slightly above the previous state of the art (Wang et al., 2018).”
Denoising Distantly Supervised Open-Domain Question Answering (2018, methodology): “Therefore, Wang et al. (2018b) propose to further explicitly aggregate evidence from across different paragraphs to re-rank extracted answers.”
Ranking Paragraphs for Improving Answer Recall in Open-Domain Question Answering (2018, methodology): “However, most of the open-domain QA pipelines depend on traditional information retrieval systems which use TF-IDF rankings (Chen et al., 2017; Wang et al., 2017b).”
You Only Need One Model for Open-domain Question Answering (2021, background): “These score the relevance of knowledge to a given query by lexical overlaps be-tween them in a sparse representation space based on TF-IDF or BM25 (Chen et al., 2017; Wang et al., 2018; Yang et al., 2019).”
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2020, methodology): “o et al., 2016) 28.60 34.60 25.90 28.50 - AQA (Buck et al., 2018) 40.50 47.40 - - - Rˆ3 (Wang et al., 2017a) 49.00 55.30 35.30 41.70 - DSQA (Lin et al., 2018) 49.00 55.30 42.30 49.30 - Evidence Agg. (Wang et al., 2018) 57.00 63.20 42.30 49.60 - BERT (Xiong et al., 2020) 57.10 61.90 40.40 46.10 - WKL”
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model (2019, methodology): “We follow previous work (Chen et al., 2017; Wang et al., 2018a) and retrieve context paragraphs with information retrieval systems.”
A Co-Matching Model for Multi-choice Reading Comprehension (2018, methodology): “In the future, we will adapt the idea of co-matching and hierarchical aggregation to the standard open-domain QA setting for answer candidate reranking (Wang et al., 2017).”
Multi-Passage Machine Reading Comprehension with Cross-Passage Answer Verification (2018, methodology): “We tune the hyper-parameters according to the Model ROUGE-L BLEU-1 FastQA Ext (Weissenborn et al., 2017) 33.67 33.93 Prediction (Wang and Jiang, 2016) 37.33 40.72 ReasoNet (Shen et al., 2017) 38.81 39.86 R-Net (Wang et al., 2017c) 42.89 42.22 S-Net (Tan et al., 2017) 45 Two simple yet effective tech”
Automated Fact Checking: Task Formulations, Methods and Future Directions (2018, methodology): “Wang et al. (2017) and Rashkin et al. (2017) expect as output multiclass labels following the deﬁnitions by the journalists over a multi-point scale but ignoring the ordering among them.”