Asking Clarifying Questions in Open-Domain Information-Seeking Conversations

By Mohammad Aliannejadi, Hamed Zamani, Fabio A. Crestani, W. Bruce Croft. 2019

419 citations recorded by the source corpus.

Users often fail to formulate their complex information needs in a single query. As a consequence, they may need to scan multiple result pages or reformulate their queries, which may be a frustrating experience. Alternatively, systems can improve user satisfaction by proactively asking questions of the users to clarify their information needs. Asking clarifying questions is especially important in conversational systems since they can only return a limited number of (often only one) result(s). In this paper, we formulate the task of asking clarifying questions in open-domain information-seeking conversational systems. To this end, we propose an offline evaluation methodology for the task and collect a dataset, called Qulac, through crowdsourcing. Our dataset is built on top of the TREC Web Track 2009-2012 data and consists of over 10K question-answer pairs for 198 TREC topics with 762 facets. Our experiments on an oracle model demonstrate that asking only one good question leads to over 170% retrieval performance improvement in terms of P@1, which clearly demonstrates the potential impact of the task. We further propose a retrieval framework consisting of three components: question retrieval, question selection, and document retrieval. In particular, our question selection model takes into account the original query and previous question-answer interactions while selecting the next question. Our model significantly outperforms competitive baselines. To foster research in this area, we have made Qulac publicly available.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

AmbigQA: Answering Ambiguous Open-domain Questions (2020, background): “Aliannejadi et al. (2019) investigate using clariﬁcation questions to iteratively reﬁne intents of simple queries where the information need is not immediately apparent (e.g., single keywords like dinosaur ).”
Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback (2020, background): “…have been studied in multiple domains to resolve ambiguity caused by speech recognition failure (Stoyanchev et al., 2014), recover missing information in question answering (Rao and Daum´e III, 2018) or clarify information needs in open-domain information-seeking (Aliannejadi et al., 2019).”
Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Search (2020, methodology): “rics. We finally report and discuss the results. 4.1 Data To evaluate our model, we use the recently proposed dataset for asking clarifying questions in information seeking conversations, called Qulac[1]. Qulac was collected through crowdsourcing based on the topics in the TREC Web Track 2009-2012 [”
MIMICS: A Large-Scale Data Collection for Search Clarification (2020, background): “To the best of our knowledge, Qulac3 [1] is the only public dataset that focuses on search clarification.”
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, background): “Second, when the question is classified as unanswerable due to ambiguity or insufficient background knowledge, the conversational OpenQA system needs to generate a followup question [149].”