Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension

By Minjoon Seo, Tom Kwiatkowski, Ankur P. Parikh, Ali Farhadi, Hannaneh Hajishirzi. 2018

60 citations recorded by the source corpus.

We formalize a new modular variant of current question answering tasks by enforcing complete independence of the document encoder from the question encoder. This formulation addresses a key challenge in machine comprehension by building a standalone representation of the document discourse. It additionally leads to a significant scalability advantage since the encoding of the answer candidate phrases in the document can be pre-computed and indexed offline for efficient retrieval. We experiment with baseline models for the new task, which achieve a reasonable accuracy but significantly underperform unconstrained QA models. We invite the QA research community to engage in Phrase-Indexed Question Answering (PIQA, pika) for closing the gap. The leaderboard is at: nlp.cs.washington.edu/piqa

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Learning to Retrieve Passages without Supervision (2021, background): “Phrase-based retrieval (Seo et al., 2018, 2019) eliminates the need for a reader during inference, as it directly retrieves the answer span given a query.”
Learning Dense Representations of Phrases at Scale (2020, methodology): “Following previous work on phrase or span representations (Lee et al., 2017; Seo et al., 2018), we ﬁrst apply a pre-trained language model M p to obtain contextualized word representations for each passage token: h 1 , . . . , h m ∈ R d .”
Synthesizer: Rethinking Self-Attention for Transformer Models (2020, background): “In reality, this is computationally prohibitive for many practical real-world applications (Seo et al., 2018).”
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them (2021, methodology): “Real-time ODQA Systems prioritizing fast runtime over accuracy are sometimes referred to as real-time QA systems (Seo et al., 2018).”
Differentiable Reasoning over a Virtual Knowledge Base (2020, background): “One option adopted by previous researchers (Seo et al., 2018) is to ﬁne-tune BERT on SQuAD (Rajpurkar et al., 2016).”
Contextualized Sparse Representations for Real-Time Open-Domain Question Answering (2019, methodology): “More recently, an alternative approach formulates the task as an end-to-end phrase retrieval problem by encoding and indexing every possible text span in a dense vector ofﬂine (Seo et al., 2018).”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, methodology): “Bi-encoder QA has been used for efficient opendomain QA via phrase retrieval (Seo et al., 2018, 2019; Lee et al., 2020, 2021), but its lower accuracy compared to cross-encoder QA has previously been viewed as a drawback.”
PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text (2019, background): “An alternative approach is “phrase-indexed” QA (Seo et al., 2018), where embedded phrases in a document are indexed and searched over.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “PIQA (phrase-indexed question answering) [182] presents the first approach that leverages dense phrase index for question answering.”
ReQA: An Evaluation for End-to-End Answer Retrieval Models (2019, methodology): “Finally, Seo et al. (2018) construct a phrase-indexed question answering challenge that is similar to ReQA in requiring the question and the answer be encoded separately of one another.”
Reasoning Over Virtual Knowledge Bases With Open Predicate Relations (2021, methodology): “PIQA (Seo et al., 2018) and DrKIT (Dhingra et al., 2020) build mention-level indexes using a pretrained encoder.”
Contextualized Sparse Representation with Rectified N-Gram Attention for Open-Domain Question Answering (2019, methodology): “In this work, we are interested in learning query-agnostic sparse phrase representations which enables the precomputation of re-usable phrase representations (Seo et al., 2018).”
Simple and Efficient ways to Improve REALM (2021, methodology): “Recently, dense retrieval methods (Seo et al., 2018, 2019; Das et al., 2019; Karpukhin et al., 2020) have improved over sparse retrievers like BM25 (Robertson and Zaragoza, 2009) and made training these systems end-to-end by leveraging approximate MIPS search (Shrivastava and Li, 2014).”
Multi-Hop Paragraph Retrieval for Open-Domain Question Answering (2019, background): “Seo et al. (2018) proposed phrase-indexed QA (PI-QA), a new formulation of the QA task that requires the independent encoding of answers and questions.”
Phrase Retrieval Learns Passage Retrieval, Too (2021, methodology): “Following previous work (Seo et al., 2018, 2019), we use the term ‘phrase’ to denote any contiguous text segment up to L words (including single words), which is not necessarily a linguistic phrase and we take phrases up to length L = 20 .”
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, methodology): “Some works [15], [16], [36], [114] propose”