Revealing the Importance of Semantic Retrieval for Machine Reading at Scale

By Yixin Nie, Songhe Wang, Mohit Bansal. 2019

147 citations recorded by the source corpus.

Machine Reading at Scale (MRS) is a challenging task in which a system is given an input query and is asked to produce a precise output by “reading” information from a large knowledge base. The task has gained popularity with its natural combination of information retrieval (IR) and machine comprehension (MC). Advancements in representation learning have led to separated progress in both IR and MC; however, very few studies have examined the relationship and combined design of retrieval and comprehension at different levels of granularity, for development of MRS systems. In this work, we give general guidelines on system design for MRS by proposing a simple yet effective pipeline system with special consideration on hierarchical semantic retrieval at both paragraph and sentence level, and their potential effects on the downstream task. The system is evaluated on both fact verification and open-domain multihop QA, achieving state-of-the-art results on the leaderboard test sets of both FEVER and HOTPOTQA. To further demonstrate the importance of semantic retrieval, we present ablation and analysis studies to quantify the contribution of neural retrieval modules at both paragraph-level and sentence-level, and illustrate that intermediate semantic retrieval modules are vital for not only effectively filtering upstream information and thus saving downstream computation, but also for shaping upstream data distribution and providing better data for downstream modeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, methodology): “Notably, our method achieves 14.5 F1 and 14.0 EM gains compared to state-of-the-art Semantic Retrieval (Nie et al., 2019) and 10.9 F1 gains over the concurrent Transformer-XH model (Zhao et al., 2020) on full wiki.”
ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, background): “Use of hyperlink structure in Wikipedia or performing entity linking to establish hyperlinked mentions, similar to our approach here, has been previously explored in multi-hop open domain question answering (Asai et al., 2020; Nie et al., 2019).”
Adaptive Information Seeking for Open-Domain Question Answering (2021, background): “As an extension, some approaches further consider the relations be-tween passages through hyperlinks or entity links and extend evidence with the linked neighbor passages (Nie et al., 2019; Das et al., 2019b; Zhao et al., 2020).”
Unsupervised Question Decomposition for Question Answering (2020, methodology): “We treat yes and no predictions as spans within the passage (prepended to each paragraph), as in Nie et al. (2019) on H OTPOT QA.”
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval (2020, methodology): “GoldEn Retriever (Qi et al., 2019) 37.9 48.6 30.7 64,2 18.9 39.1 Semantic Retrieval (Nie et al., 2019) 46.5 58.8 39.9 71.5 26.6 49.2 Transformer-XH (Zhao et al., 2020) 51.6 64.1 40.9 71.4 26.1 51.3 HGN (Fang et al., 2019) 56.7 69.2 50.0 76.4 35.6 59.9 DrKIT (Dhingra et al., 2020) 42.1 51.7 37.1…”
Retrieve, Rerank, Read, then Iterate: Answering Open-Domain Questions of Arbitrary Complexity from Text (2020, methodology): “Secondly, previous work often assumes access to non-textual metadata such as knowledge bases, entity linking, and Wikipedia hyperlinks when retrieving supporting facts (Nie et al., 2019; Feld-man and El-Yaniv, 2019; Zhao et al., 2019; Asai et al., 2020; Dhingra et al., 2020; Zhao et al., 2020).”
Answer Complex Questions: Path Ranker Is All You Need (2021, background): “Since Chen et al. [4] has firstly proposed open-domain QA task, many papers have made improvement on the retriever (Nie et al., 2019 [21]; Guu et al, 2020 [10]) and the reader (Ni et al., 2019 [20]) module in the basic pipeline.”
Simple and Efficient ways to Improve REALM (2021, methodology): “retrieval like BM25 or TF-IDF for retrieving passages/documents (Chen et al., 2017; Yang et al., 2019; Nie et al., 2019; Min et al., 2019b).”
Highly Parallel Autoregressive Entity Linking with Discriminative Correction (2021, unknown): “, 2020), question answering (Nie et al., 2019; Asai et al., 2020; De Cao et al., 2019), dialog modeling (Dinan et al.”
Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention (2020, methodology): “Many previous FEVER systems first retrieve the evidence sentences for the claim and then reason verify it (Nie et al., 2019; Zhou et al., 2019).”
T RANSFORMER -XH: M ULTI - HOP Q UESTION A N - SWERING WITH E X TRA H OP A TTENTION (2019, methodology): “The main baselines include Cognitive QA (CogQA, Ding et al. (2019)) and Semantic Retrieval MRS (SR-MRS, Nie et al. (2019)).”
A Simple Yet Strong Pipeline for HotpotQA (2020, methodology): “In the fullwiki setting, we use the paragraphs SR-MRS (Nie et al., 2019) as a starting point.”
Fine-grained Fact Verification with Kernel Graph Attention Network (2019, methodology): “The recently launched FEVER shared task 1.0 is hosted as a competition on Codalab1 with a blind test set and has drawn lots of attention from NLP community.”
DDRQA: Dynamic Document Reranking for Open-domain Multi-hop Question Answering (2020, methodology): “Following existing studies (Asai et al., 2020; Nie et al., 2019; Nishida et al., 2019), we evaluate the paragraph-level retrieval accuracy using the Paragraph Exact Match (EM) metric, which compares the top 2 paragraphs with the gold supporting paragraphs.”