Interpretation of Natural Language Rules in Conversational Machine Reading

By Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktaschel, Mike Sheldon, Guillaume Bouchard, Sebastian Riedel. 2018

174 citations recorded by the source corpus.

Most work in machine reading focuses on question answering problems where the answer is directly expressed in the text to read. However, many real-world question answering problems require the reading of text not because it contains the literal answer, but because it contains a recipe to derive an answer together with the reader’s background knowledge. One example is the task of interpreting regulations to answer “Can I...?” or “Do I have to...?” questions such as “I am working in Canada. Do I have to carry on paying UK National Insurance?” after reading a UK government website about this topic. This task requires both the interpretation of rules and the application of background knowledge. It is further complicated due to the fact that, in practice, most questions are underspecified, and a human assistant will regularly have to ask clarification questions such as “How long have you been working abroad?” when the answer cannot be directly derived from the question and text. In this paper, we formalise this task and develop a crowd-sourcing strategy to collect 37k task instances based on real-world rules and crowd-generated questions and scenarios. We analyse the challenges of this task and assess its difficulty by evaluating the performance of rule-based and machine-learning baselines. We observe promising results when no background knowledge is necessary, and substantial room for improvement whenever background knowledge is needed.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, background): “Yes/No questions make up a subset of the reading comprehension datasets CoQA (Reddy et al., 2018), QuAC (Choi et al., 2018), and Hot-PotQA (Yang et al., 2018), and are present in the ShARC (Saeidi et al., 2018) dataset.”
CoQA: A Conversational Question Answering Challenge (2018, background): “Concurrently, Saeidi et al. (2018) created a conversational QA dataset for regulatory text such as tax and visa regulations.”
QuAC: Question Answering in Context (2018, background): “Concurrent to our work, Saeidi et al. (2018) proposed a task of generating and answering yes/no questions for rule focused text (such as trafﬁc laws) by interacting with a user through dialog.”
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020, background): “…reasoning over multiple sentences or documents (Welbl et al., 2018; Yang et al., 2018), questions requiring rule interpretation or context awareness (Saeidi et al., 2018; Choi et al., 2018; Reddy et al., 2019), limiting annotator passage exposure by sourcing questions ﬁrst (Kwiatkowski et al.,…”
Reasoning Over Paragraph Effects in Situations (2019, methodology): “ShARC shares the same goal of understanding causes and effects (in terms of speciﬁed rules), but frames it as a dialogue where the sys-tem has to also generate questions to gain complete information.”
Unsupervised Commonsense Question Answering with Self-Talk (2020, background): “ing (Saeidi et al., 2018; Pan et al., 2019), simplifying questions to make them more easily answerable (Buck et al.”
PubMedQA: A Dataset for Biomedical Research Question Answering (2019, methodology): “Yes/No QA: Datasets such as HotpotQA (Yang et al., 2018), Natural Questions (Kwiatkowski et al., 2019), ShARC (Saeidi et al., 2018) and BioASQ (Tsatsaronis et al., 2015) contain yes/no questions as well as other types of questions.”
Dialog Inpainting: Turning Documents into Dialogs (2022, background): “Several manually collected conversational question answering datasets have been proposed to address the scarcity of highquality training data (Choi et al., 2018; Reddy et al., 2019; Dinan et al., 2018; Saeidi et al., 2018; Dalton et al., 2019; Campos et al., 2020; Dalton et al., 2020; Qu et al., 202”
A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering (2020, unknown): “Saeidi et al. [49] frame the task of statutory reasoning as a dialog between a user and a dialog agent.”
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, background): “7 Multiple choices No additional context ShARC [133] Others 32.”
CONVERSATIONAL MACHINE COMPREHENSION (2019, background): “Recent datasets (Choi et al., 2018; Reddy et al., 2019; Elgohary et al., 2018; Saeidi et al., 2018) enabled studying it in the textual setting, where the information source used to answer questions is a given article.”
Evaluating Rewards for Question Generation Models (2019, background): “…about a document in natural language is a crucial aspect of the effort to automatically process natural language data, enabling machines to ask clariﬁcation questions (Saeidi et al., 2018), become more robust to queries (Yu et al., 2018), and to act as automatic tutors (Heilman and Smith, 2010).”
Cross-Policy Compliance Detection via Question Answering (2021, methodology): “While the policies and scenarios in the dataset are taken from ShARC (Saeidi et al., 2018), we augment them with expression trees and answers to each question for all the scenarios and policies to create the Question Answering for Policy Compliance (QA4PC ) dataset.”
Open-Domain Question Answering Goes Conversational via Question Rewriting (2020, background): “This task is commonly referred to as conversational (interactive or sequential) QA (Webb, 2006; Saeidi et al., 2018; Reddy et al., 2019).”
On Making Reading Comprehension More Comprehensive (2019, background): “There are two datasets that just scratch the surface of this kind of reading: ShARC (Saeidi et al., 2018) requires reading rules and applying them to questions asked by users, though its format is not standard reading comprehension; and ROPES (Lin et al., 2019), which requires reading descriptions o”
Implicit Discourse Relation Classification: We Need to Talk about Evaluation (2020, background): “Understanding discourse relations in natural language text is crucial to end tasks involving larger context, such as question-answering (Jansen et al., 2014) and conversational systems grounded on documents (Saeidi et al., 2018; Feng et al., 2020).”
Neural Databases (2020, methodology): “More recently, systems such as BREAK [57] and ShARC [41] have trained models to translate a natural language query into a sequence of relational operators (or variants thereof).”
Ditch the Gold Standard: Re-evaluating Conversational Question Answering (2021, methodology): “With recent development of large-scale English CQA datasets (Choi et al., 2018; Saeidi et al., 2018; Reddy et al., 2019; Campos et al., 2020), rapid progress has been made in better modeling of conversational QA systems.”
FlowQA: Grasping Flow in History for Conversational Machine Comprehension (2018, background): “ften framed as a semantic parsing problem. Recent 12This domain contains substantially smaller training data (˘20K QA pairs). 8 datasets (Choi et al., 2018; Reddy et al., 2018; Elgohary et al., 2018; Saeidi et al., 2018) enabled studying it in the textual setting, where the information source used t”