Ask the Right Questions: Active Question Reformulation with Reinforcement Learning

By Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Andrea Gesmundo, Neil Houlsby, Wojciech Gajewski, Wei Wang. 2017

177 citations recorded by the source corpus.

We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. The reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!. The agent outperforms a state-of-the-art base model, playing the role of the environment, and other benchmarks. We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “While there are early works focused on models that could interface with external tools, they employ ad-hoc tool-dependent architectures (Adolphs et al., 2022; Buck et al., 2018; Nogueira and Cho, 2017; Zhong et al., 2018).”
Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering (2017, methodology): “…et al., 2017a;b), a reading comprehension model with gated-attention; BiDAF (Seo et al., 2017), a RC model with bidirectional attention ﬂow; AQA (Buck et al., 2017), a reinforced system learning to aggregate the answers generated by the re-written questions; R 3 (Wang et al., 2017), a…”
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering (2019, background): “Perhaps the most related to our work is Active Question Answering (AQA) (Buck et al., 2018), which use reformulation of the natural language query to improve the performance of a BiDAF reader model on S EARCHQA .”
Denoising Distantly Supervised Open-Domain Question Answering (2018, methodology): “(3) AQA (Buck et al., 2017), a reinforced system learning to re-write questions and aggregate the answers generated by the re-written questions; (4) R3 (Wang et al.”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “• Query Reformulation Query reformulation [18, 82] is commonly used in information retrieval [90] and question answering tasks [10, 136], which aim to elicit more relevant texts (documents or answers) by expanding the input query with related query terms [42] or generating paraphrases.”
The Web as a Knowledge-Base for Answering Complex Questions (2018, methodology): “Paraphrasing for better interaction with a QA model has been recently suggested by Buck et al. (2017) and Nogueira and Cho (2016).”
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2020, background): “AQA (Buck et al., 2018) is a reinforced system learning to re-write questions and aggregate the answers generated by the re-written questions.”
Unsupervised Commonsense Question Answering with Self-Talk (2020, background): “, 2019), simplifying questions to make them more easily answerable (Buck et al., 2018; Talmor and Berant, 2018; Perez et al., 2020), to using questions as means for other purposes such as sentence representation and sum-”
Task-Oriented Query Reformulation with Reinforcement Learning (2017, methodology): “To solve these problems, we extend the model to sequentially generate a reformulated query, as proposed by Buck et al. (2017).”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, background): “Finally, various approaches (Chen et al., 2016; Buck et al., 2018; Nogueira and Cho, 2017; Wang et al., 2018) have also proposed reformulating questions for textual QA tasks.”
Identifying Well-formed Natural Language Questions (2018, background): “Another direction is to obtain a paraphrase of the given query that is a grammatical natural language question, and then analyze that paraphrase to extract the required information (Nogueira and Cho, 2017; Buck et al., 2018).”
Boosting Search Engines with Interactive Agents (2021, neutral): “Nogueira & Cho (2017) and Buck et al. (2018) use RL-trained agents to seek good answers by reformulating questions with seq2seq models.”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, background): “(2017); Buck et al. (2018); Talmor and Berant (2018); Min et al.”
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering (2019, positive): “A few recent works learn to reformulate the query using task specific reward such as document recall or performance on a QA task (Nogueira and Cho, 2017; Buck et al., 2018; Das et al., 2019).”
Improving the Robustness of Question Answering Systems to Question Paraphrasing (2019, methodology): “Buck et al. (2017) formulated QA as a reinforcement learning problem and introduced a paraphrasing agent trained to paraphrase a question to one that was able to get the best answer from the QA model.”
A Joint Model for Question Answering and Question Generation (2017, methodology): “Buck et al. (2017) used policy gradient with a QA reward to train a sequence-to-sequence paraphrase model to reformulate questions in an existing QA dataset (Dunn et al., 2017).”
Deep Reinforced Query Reformulation for Information Retrieval (2020, methodology): “Buck et al. proposed a question reformulation model based on seq2seq trained using reinforcement learning for the QA task [3].”
Multi-agent query reformulation: Challenges and the role of diversity (2019, methodology): “We build upon the works of Nogueira & Cho (2017) and Buck et al. (2018b).”