Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph

By Amrita Saha, Vardaan Pahuja, Mitesh M. Khapra, Karthik Sankaranarayanan, A. P. Sarath Chandar. 2018

212 citations recorded by the source corpus.

While conversing with chatbots, humans typically tend to ask many questions, a significant portion of which can be answered by referring to large-scale knowledge graphs (KG). While Question Answering (QA) and dialog systems have been studied independently, there is a need to study them closely to evaluate such real-world scenarios faced by bots involving both these tasks. Towards this end, we introduce the task of Complex Sequential QA which combines the two tasks of (i) answering factual questions through complex inferencing over a realistic-sized KG of millions of entities, and (ii) learning to converse through a series of coherently linked QA pairs. Through a labor intensive semi-automatic process, involving in-house and crowdsourced workers, we created a dataset containing around 200K dialogs with a total of 1.6M turns. Further, unlike existing large scale QA datasets which contain simple questions that can be answered from a single tuple, the questions in our dialogs require a larger subgraph of the KG. Specifically, our dataset has questions which require logical, quantitative, and comparative reasoning as well as their combinations. This calls for models which can: (i) parse complex natural language questions, (ii) use conversation context to resolve coreferences and ellipsis in utterances, (iii) ask for clarifications for ambiguous queries, and finally (iv) retrieve relevant subgraphs of the KG to answer such questions. However, our experiments with a combination of state of the art dialog and QA models show that they clearly do not achieve the above objectives and are inadequate for dealing with such complex real world settings. We believe that this new dataset coupled with the limitations of existing models as reported in this paper should encourage further research in Complex Sequential QA.

Subfields: Artificial Intelligence, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

CoQA: A Conversational Question Answering Challenge (2018, background): “…a hand-written grammar to create artificial questions (Weston et al., 2016; Welbl et al., 2018), paraphrasing artificial questions to natural questions (Saha et al., 2018; Talmor and Berant, 2018), or, in our case, letting humans ask natural questions (Rajpurkar et al., 2016; Nguyen et al., 2016).”
QuAC: Question Answering in Context (2018, background): “CoQA (Reddy et al., 2018) CSQA (Saha et al., 2018) CQA (Talmor and Berant, 2018) SQA (Iyyer et al., 2017) NarrativeQA (Kocisk´y et al., 2017) TriviaQA (Joshi et al., 2017) SQuAD 2.0 (Rajpurkar et al., 2018) MS Marco (Nguyen et al., 2016) NewsQA (Trischler et al., 2016) Table 1: Comparison of the QUA”
Towards a Better Metric for Evaluating Question Generation Systems (2018, background): “…Answering (QA) (Rajpurkar et al., 2016; Nguyen et al., 2016; Joshi et al., 2017; Saha et al., 2018a) knowledge base driven QA (Bordes et al., 2015; Saha et al., 2018b) and Visual QA (Antol et al., 2015; Johnson et al., 2017) has enabled the development of end-to-end supervised models for QA.”
Does Gender Matter? Towards Fairness in Dialogue Systems (2019, background): “Conversational question answering agents provide users with the information they want to find (Saha et al., 2018).”
Interpretation of Natural Language Rules in Conversational Machine Reading (2018, background): “In this space, previous approaches have been looking primarily at QA dialogs about images (Das et al., 2017) and knowledge graphs (Saha et al., 2018; Iyyer et al., 2017).”
Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing (2019, methodology): “More detailed numbers are provided in the appendix C. HRED+KVmem (Saha et al., 2018) is a encoder-decoder model with key-value memory network (Miller et al., 2016) to directly produce answers.”
Few-Shot NLG with Pre-Trained Language Model (2019, background): “Some examples are task-oriented dialog, question answering (He et al. 2017; Su et al. 2016; Saha et al. 2018; Yin et al. 2016) and interdisciplinary applications such as medicine (Hasan and Farri 2019; Cawsey, Webber, and Jones 1997) and health-care (Hasan and Farri 2019; DiMarco et al. 2007),…”
Teaching Small Language Models to Reason (2022, background): “We do not benchmark on the CSQA dataset (Saha et al., 2018) as it has 200,000 examples, which is too costly to infer as stated previously.”
Leveraging Domain Context for Question Answering Over Knowledge Graph (2019, background): “The difficulty issues lie at not only the vague question description but also a variety of entity and relationship types within the knowledge graph [23, 31].”
Making Large Language Models Better Reasoners with Alignment (2023, unknown): “Specifically, the task accuracy and the assessment accuracy exhibit a strong positive correlation, with Pearson Correlation Coefficients of 0.93 and 0.98 at GSM8K and ECQA, respectively.”
Continuous and Interactive Factual Knowledge Learning in Veriﬁcation Dialogues (2020, background): “There are also works on KB question-answering (KBQA) [22, 7, 4, 42, 41], open-vocabulary semantic parsing [15, 47], information extraction [43, 2], and KB updating using facts extracted from the Web [35].”
A dataset and baselines for sequential open-domain question answering (2018, background): “Saha et al. (2018) study the problem of sequential question answering, and introduce a dataset for the task.”
CONVERSATIONAL MACHINE COMPREHENSION (2019, background): “Sequential question answering has been studied in the knowledge base setting (Iyyer et al., 2017; Saha et al., 2018; Talmor & Berant, 2018), often framed as a semantic parsing problem.”
Asking Clarification Questions in Knowledge-Based Question Answering (2019, background): “Unlike dataset (Saha et al., 2018) that constructs clariﬁca-tion questions base on predicate-independent templates, our clariﬁcation questions are predicate-aware and more diverse.”
Ditch the Gold Standard: Re-evaluating Conversational Question Answering (2021, methodology): “, 2018), utilizing structured knowledge base (Saha et al., 2018; Guo et al., 2018), as well as CQA in other modalities (Das et al.”