Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering

By Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, Bing Xiang. 2019

265 citations recorded by the source corpus.

BERT model has been successfully applied to open-domain QA tasks. However, previous work trains BERT by viewing passages corresponding to the same question as independent training instances, which may cause incomparable scores for answers from different passages. To tackle this issue, we propose a multi-passage BERT model to globally normalize answer scores across all passages of the same question, and this change enables our QA model find better answers by utilizing more passages. In addition, we find that splitting articles into passages with the length of 100 words by sliding window improves performance by 4%. By leveraging a passage ranker to select high-quality passages, multi-passage BERT gains additional 2%. Experiments on four standard benchmarks showed that our multi-passage BERT outperforms all state-of-the-art models on all benchmarks. In particular, on the OpenSQuAD dataset, our model gains 21.4% EM and 21.5% F1 over all non-BERT models, and 5.8% EM and 6.5% F1 over BERT-based models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, background): “2 In DPR, passages and questions are represented as dense vector representations, computed using two BERT networks.”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “In open-domain QA, models can extract a span of text from retrieved documents as the answer (Chen et al., 2017; Clark & Gardner, 2018; Wang et al., 2019; Karpukhin et al., 2020), a method inspired by reading comprehension (Richardson, 2013; Rajpurkar et al., 2016).”
Big Bird: Transformers for Longer Sequences (2020, methodology): “…based tasks – as a seq2seq model for translation [92], summarization [66], generation [15], etc. or as a standalone encoders for sentiment analysis [84], POS tagging [65], machine reading comprehension [94], etc. – and it is known to vastly outperform previous sequence models like LSTM [37].”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, background): “Finally, from entries (1) and (5), we see that the shared normalization is essential in open-domain QA, which aligns with the ﬁndings in Wang et al. (2019).”
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, methodology): “…our reader jointly learns to predict the plausibility and answer the question, and moreover, fully leverages the self-attention mechanism across the concatenated paragraphs in the retrieved reasoning paths; this paragraph interaction is crucial for multi-hop reasoning (Wang et al., 2019a).”
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering (2019, background): “(1) P AR R EADER reads each passage in parallel and predicts a candidate span from the passage and an answerable score (Min et al., 2019b; Wang et al., 2019).”
R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, methodology): “Contrary to belief based on the results on different datasets (Yang et al., 2019; Wang et al., 2019b; Izacard and Grave, 2021), we found the extractive reader can also benefit from larger input sizes, both in training and test time.”
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, background): “Extractive readers (Clark and Gardner, 2018, de Masson d’Autume et al., 2019, Wang et al., 2019, Guu et al., 2020, Karpukhin et al., 2020) extract a span from a retrieved document to produce an answer.”
A Memory Efficient Baseline for Open Domain Question Answering (2020, methodology): “l called the reader to produce the answer. In case where no gold spans are available,Clark and Gardner(2018) proposed a loss function based on global normalization over all the mentions of the answer.Wang et al. (2019) applied this approach to BERT pre-trained models, whileMin et al.(2019) introduce”
Learning Dense Representations of Phrases at Scale (2020, background): “It can easily adapt to passages and sentences (Yang et al., 2019; Wang et al., 2019).”
Adaptive Information Seeking for Open-Domain Question Answering (2021, background): “…(Qi et al., 2020) - - - - - - 66.3 79.9 57.2 82.6 43.1 69.8 MUPPET (Feldman and El-Yaniv, 2019) 31.1 40.4 17.0 47.7 11.8 27.6 30.6 40.3 16.7 47.3 10.9 (Wang et al., 2019b) 53.0 60.9 100 DPR (Karpukhin et al., 2020) 29.8 - 100 BM25+DPR (Karpukhin et al., 2020) 36.7 - 100 Multi-step Reasoner (Das…”
Relevance-guided Supervision for OpenQA with ColBERT (2020, methodology): “For instance, Wang et al. (2019) and Karpukhin et al. (2020) consider only passages highly-ranked by BM25 for q as candidates, where the answer string can be a more reliable signal.”
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model (2019, background): “For instance, the new state-of-the-art for WNLI (Wang et al., 2019a), ReCoRD (Zhang et al., 2018) and SWAG (Zellers et al., 2018) is achieved by pretrained models.”
Multi-Task Retrieval for Knowledge-Intensive Tasks (2021, methodology): “Preprocessing Starting from the raw KILT data, we split each Wikipedia article into disjoint 100-token chunks which form our basic retrieval units, following the approach of Wang et al. (2019) and Karpukhin et al. (2020).”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “…passages from a ﬁrst-stage retriever, BERT-based rerankers have recently been applied to retrieval-based question answering and search-related tasks (Wang et al., 2019; Nogueira and Cho, 2019; Nogueira et al., 2019b; Yan et al., 2019), and yield substantial improvements over the traditional…”
Open-Domain Question Answering (2020, background): “We will further discuss several challenges and techniques arising in this area, including multi-passage training (Clark and Gardner, 2018; Wang et al., 2019), passage reranking (Wang et al.”
What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams (2020, background): “Afterwards, researchers have introduced more sophisticated models, which either aggregate all informative evidence (Clark and Gardner 2018; Wang et al. 2019) or filter out those irrelevant retrieved texts (Wang et al.”
CogLTX: Applying BERT to Long Texts (2020, methodology): “A straightforward solution for long texts is sliding window [50], processing continuous 512-token spans by BERT.”