Task-Oriented Query Reformulation with Reinforcement Learning

By Rodrigo Nogueira, Kyunghyun Cho. 2017

187 citations recorded by the source corpus.

Search engines play an important role in our everyday lives by assisting us in finding the information we need. When we input a complex query, however, results are often far from satisfactory. In this work, we introduce a query reformulation system based on a neural network that rewrites a query to maximize the number of relevant documents returned. We train this neural network with reinforcement learning. The actions correspond to selecting terms to build a reformulated query, and the reward is the document recall. We evaluate our approach on three datasets against strong baselines and show a relative improvement of 5-20% in terms of recall. Furthermore, we present a simple method to estimate a conservative upper-bound performance of a model in a particular environment and verify that there is still large room for improvements.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “While there are early works focused on models that could interface with external tools, they employ ad-hoc tool-dependent architectures (Adolphs et al., 2022; Buck et al., 2018; Nogueira and Cho, 2017; Zhong et al., 2018).”
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering (2019, methodology): “Instead of using such automatic relevance feedback, Nogueira & Cho (2017) train a query reformulation model to maximize the recall of a IR system using RL and Pfeiffer et al. (2018) showed that query reﬁnement is effective for IR in bio-medical domain.”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “Discrete Prompt Search [52, 125] Query reformulation [90, 90] Reformulate the input into a query form Query reformulation commonly focuses on information extraction and question answering tasks, while prompt learning can be applied to a variety of NLP tasks”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “These include: 1) generation of good search queries [22, 59] and reformulation [37, 51], 2) strategic exploration for navigating through the website [55, 56, 29], 3) robust language understanding for textual state and action spaces [3, 7, 17, 44], and 4) long-term”
Internet-Augmented Dialogue Generation (2021, background): “plied directly to improve the search task itself, for example using reinforcement learning (Nogueira and Cho, 2017).”
Unsupervised Question Decomposition for Question Answering (2020, background): “Weakly supervised question generation often uses reinforcement learning (Nogueira & Cho, 2017; Wang & Lake, 2019; Strub et al., 2017; Das et al., 2017; Liang et al., 2018), where an unsupervised initialization can greatly mitigate the issues of exploring from scratch (Jaderberg et al., 2017).”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, background): “Learning Search Expansion Other research addresses expanding the document set available to a QA system, either in the form of web navigation (Nogueira and Cho, 2016), or via query reformulation techniques, which often use neural reinforcement learning (Narasimhan et al., 2016; Nogueira and Cho, 2017”
Ranking Sentences for Extractive Summarization with Reinforcement Learning (2018, background): “, 2017) or to inject task-specific constraints (Zhang and Lapata, 2017; Nogueira and Cho, 2017).”
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning (2017, background): “In a recent follow-up Nogueira and Cho [2017] improve document retrieval with an approach inspired by relevance feedback in combination with RL.”
Generation-Augmented Retrieval for Open-Domain Question Answering (2020, methodology): “There are recent or concurrent studies (Nogueira and Cho, 2017; Zaiem and Sadat, 2019; Yu et al., 2020; Vakulenko et al., 2020; Lin et al., 2020) that reformulate queries with generation models for other retrieval tasks.”
Identifying Well-formed Natural Language Questions (2018, background): “Another direction is to obtain a paraphrase of the given query that is a grammatical natural language question, and then analyze that paraphrase to extract the required information (Nogueira and Cho, 2017; Buck et al., 2018).”
Boosting Search Engines with Interactive Agents (2021, neutral): “Nogueira & Cho (2017) and Buck et al. (2018) use RL-trained agents to seek good answers by reformulating questions with seq2seq models.”
Search Engine Guided Non-Parametric Neural Machine Translation (2017, background): “Recently, Nogueira and Cho (2017) proposed task-oriented query reformulation in which a neural network is trained to use a black-box search engine to maximize the recall of relevant documents, which can be integrated into the proposed TM-NMT.”
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering (2019, positive): “A few recent works learn to reformulate the query using task specific reward such as document recall or performance on a QA task (Nogueira and Cho, 2017; Buck et al., 2018; Das et al., 2019).”
Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension (2018, background): “Hence other recent approaches propose to jointly learn them via reinforcement learning (Choi et al., 2017; Wang et al., 2018; Nogueira and Cho, 2017) or cascades (Swayamdipta et al.”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, background): “The majority of RL research in NLP has focused on learning from online feedback (Sokolov et al., 2016; He et al., 2016; Li et al., 2016; Bahdanau et al., 2017; Nguyen et al., 2017; Nogueira and Cho, 2017; Lam et al., 2018).”
Interactive Machine Comprehension with Information Seeking Agents (2019, background): “Broadly speaking, our work is also linked to the query reformulation (QR) task in information retrieval literature (Nogueira and Cho, 2017).”
Conversational Recommender System (2018, methodology): “Deep RL has been applied for better sequential decision making in various domains, including End-to-End dialogue systems using deep RL for information access [5] [32], information extraction [16], query reformulation [17], real time ads bidding [2].”