Open-Domain Question Answering Goes Conversational via Question Rewriting

By R. Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen G. Pulman, Srinivas Chappidi. 2020

230 citations recorded by the source corpus.

We introduce a new dataset for Question Rewriting in Conversational Context (QReCC), which contains 14K conversations with 80K question-answer pairs. The task in QReCC is to find answers to conversational questions within a collection of 10M web pages (split into 54M passages). Answers to questions in the same conversation may be distributed across several web pages. QReCC provides annotations that allow us to train and evaluate individual subtasks of question rewriting, passage retrieval and reading comprehension required for the end-to-end conversational question answering (QA) task. We report the effectiveness of a strong baseline approach that combines the state-of-the-art model for question rewriting, and competitive models for open-domain QA. Our results set the first baseline for the QReCC dataset with F1 of 19.10, compared to the human upper bound of 75.45, indicating the difficulty of the setup and a large room for improvement.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Scaling Instruction-Finetuned Language Models (2022, background): “(2021) and 26 new tasks that we added in this work, including dialog data (Byrne et al., 2019; Anantha et al., 2021; Dai et al., 2022) and program synthesis data (Yasunaga and Liang, 2020; Li et al.”
Measuring Attribution in Natural Language Generation Models (2021, methodology): “Conversational QA QReCC (Anantha et al. 2021) Conversational History Retrieved Document Response”
Ditch the Gold Standard: Re-evaluating Conversational Question Answering (2021, methodology): “, 2020), as well as a few recent works focusing on conversational opendomain question answering (Adlakha et al., 2021; Anantha et al., 2021; Qu et al., 2020) Different from single-turn QA datasets (Rajpurkar et al.”
A Wrong Answer or a Wrong Question? An Intricate Relationship between Question Reformulation and Answer Selection in Conversational Question Answering (2020, methodology): “For our experiments, we adopt the same QR model architecture proposed in the previous work (Transformer++) (Anantha et al., 2020; Vakulenko et al., 2020).”
Attributed Text Generation via Post-hoc Research and Revision (2022, methodology): “We then use these metrics to study how editors perform on outputs from state-of-the-art language models, such as knowledge-intensive statements, reasoning chains (Geva et al., 2021) and dialog responses (Anantha et al., 2021).”
Task-aware Retrieval with Instructions (2022, background): “Table 1: Example instructions for Natural Questions (NQ; Kwiatkowski et al. 2019), QReCC (Anantha et al., 2021), Arguana (Wachsmuth et al., 2018), SciFact (Wadden et al., 2020) and MultiLexSum (Shen et al., 2022).”
Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, background): “Additionally, real-world dialogues often feature co-references and missing content ( i.e., anaphora) (Anantha et al., 2021), which further complicate the retrieval process (Mallen et al., 2023; Gao et al., 2023b; Liu et al., 2023b); (2) Challenges arise in reasoning over retrieved documents,…”
Dialog Inpainting: Turning Documents into Dialogs (2022, methodology): “Datasets We use four open-domain conversational QA retrieval benchmarks: OR-QuAC (Qu et al., 2020), TREC CAsT-19 (Byrne et al., 2019), TREC CAsT-20 (Dalton et al., 2020), and QReCC (Anantha et al., 2021).”
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP (2022, methodology): “…of SQuAD (Rajpurkar et al., 2016; Lee et al., 2019), the multi-hop HotPotQA (Yang et al., 2018) dataset in the open-domain “fullwiki” setting, and the conversational question answering QReCC (Anantha et al., 2020; Vakulenko et al., 2022) dataset, which we used for developing the DSP abstractions.”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, methodology): “We consider the conversational question answering task from the QReCC dataset (Anantha et al., 2021).”
CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning (2021, methodology): “We show C ON QRR outperforms existing QR models on a recent large-scale open-domain CQA dataset QReCC (Anantha et al., 2021) by over 12% and 14% for BM25 and a neural dual encoder re-triever model (Ni et al., 2021) respectively, averaging over three retrieval metrics.”