Answering Complex Open-domain Questions Through Iterative Query Generation

By Peng Qi, Xiaowen Lin, L. Mehr, Zijian Wang, Christopher D. Manning. 2019

134 citations recorded by the source corpus.

It is challenging for current one-step retrieve-and-read question answering (QA) systems to answer questions like “Which novel by the author of ‘Armada’ will be adapted as a feature film by Steven Spielberg?” because the question seldom contains retrievable clues about the missing entity (here, the author). Answering such a question requires multi-hop reasoning where one must gather information about the missing entity (or facts) to proceed with further reasoning. We present GoldEn (Gold Entity) Retriever, which iterates between reading context and retrieving more supporting documents to answer open-domain multi-hop questions. Instead of using opaque and computationally expensive neural retrieval models, GoldEn Retriever generates natural language search queries given the question and available context, and leverages off-the-shelf information retrieval systems to query for missing entities. This allows GoldEn Retriever to scale up efficiently for open-domain multi-hop reasoning while maintaining interpretability. We evaluate GoldEn Retriever on the recently proposed open-domain multi-hop QA dataset, HotpotQA, and demonstrate that it outperforms the best previously published model despite not using pretrained language models such as BERT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, methodology): “The effectiveness of arbitrary-step retrieval The existing iterative retrieval methods ﬁx the number of reasoning steps (Qi et al., 2019; Das et al., 2019; Godbole et al., 2019; Feldman & El-Yaniv, 2019), while our approach accommodates arbitrary steps of reasoning.”
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval (2020, methodology): “After we rerank the top-100 sequences from the dense retriever, our passage recall is better than the state-of-the-art Graph Recurrent Retriever, which uses BERT to process more than 500 passages.”
Differentiable Reasoning over a Virtual Knowledge Base (2020, background): “Instead, inspired by recent works (Das et al., 2019b; Qi et al., 2019), we look at the challenging sub-task of retrieving the passages required to answer the questions from a pool of 5 .”
Hierarchical Graph Network for Multi-hop Question Answering (2019, background): “, 2019), GoldEn Retriever (Qi et al., 2019), Cognitive Graph (Ding et al.”
Multi-hop Question Answering via Reasoning Chains (2019, background): “A recent line of works on open-domain multi-hop QA (Feldman and El-Yaniv, 2019; Das et al., 2019; Asai et al., 2019; Qi et al., 2019; Godbole et al., 2019) also adopt the idea of forming query in a iterative way to select the most relevant documents regarding the question.”
DDRQA: Dynamic Document Reranking for Open-domain Multi-hop Question Answering (2020, methodology): “GoldEn Re-triever (Qi et al., 2019) adopts iterative TF-IDF retrieval by generating a new query for each retrieval step.”
Answer Complex Questions: Path Ranker Is All You Need (2021, methodology): “Most recently, Qi introduces GoldEn Retriever [23], which adopts iterative TF-IDF retrieval by generating a new query for each retrieval step.”
Shepherd Pre-trained Language Models to Develop a Train of Thought: An Iterative Prompting Approach (2022, methodology): “…to queries with a single, identifiable relation/predicate (Jiang et al., 2020; Petroni et al., 2019; Zhong et al., 2021; Shin et al., 2020; Qin and Eisner, 2021), or being agnostic and insensitive to step-wise inputs (Lester et al., 2021; Li and Liang, 2021; Brown et al., 2020), and hence…”
Retrieve, Rerank, Read, then Iterate: Answering Open-Domain Questions of Arbitrary Complexity from Text (2020, methodology): “While most previous work on iterative retrieval make use of neural retrieval systems that directly accept real vectors as input, our work is similar to that of Qi et al. (2019) in using natural language search queries.”
Iteratively Prompt Pre-trained Language Models for Chain of Thought (2022, methodology): “We are also inspired by methods that iteratively retrieve knowledge from explicit knowledge sources for multi-step reasoning, such as (Xiong et al., 2021; Qi et al., 2019; Khattab et al., 2021; Mo et al., 2022b).”
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2022, methodology): “In this line of work, GoldEn (Gold Entity) retriever (Qi et al., 2019) is the closest to ours.”
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, methodology): “them into a neural MRC model [27], [32], [33], retrieving the relevant documents iteratively given a question [28], [34], [35], and training the entire OpenQA system in an end-toend manner [15], [29], [36], [37], etc.”
Answering Any-hop Open-domain Questions with Iterative Document Reranking (2020, methodology): “GoldEn Retriever [25] adopts iterative TF-IDF retrieval by generating a new query for each retrieval step.”
Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval (2021, background): “Lastly, our supervision for the ﬁrst hop shares inspiration with GoldEn [22].”
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, background): “…but interest has been growing in building multi-stage pipelines and agents that decompose complex tasks into more manageable calls to LMs in an effort to improve performance (Qi et al., 2019; Khattab et al., 2021a; Karpas et al., 2022; Dohan et al., 2022; Pourreza & Rafiei, 2023; Yao et al., 2023).”
ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data (2020, methodology): “We followed the setups in Qi et al. (2019).”