QED: A Framework and Dataset for Explanations in Question Answering

By Matthew Lamm, Jennimaria Palomaki, Chris Alberti, D. Andor, Eunsol Choi, Livio Baldini Soares, Michael Collins. 2020

83 citations recorded by the source corpus.

A question answering system that in addition to providing an answer provides an explanation of the reasoning that leads to that answer has potential advantages in terms of debuggability, extensibility, and trust. To this end, we propose QED, a linguistically informed, extensible framework for explanations in question answering. A QED explanation specifies the relationship between a question and answer according to formal semantic notions such as referential equality, sentencehood, and entailment. We describe and publicly release an expert-annotated dataset of QED explanations built upon a subset of the Google Natural Questions dataset, and report baseline models on two tasks—post- hoc explanation generation given an answer, and joint question answering and explanation generation. In the joint setting, a promising result suggests that training on a relatively small amount of QED data can improve question answering. In addition to describing the formal, language-theoretic motivations for the QED approach, we describe a large user study showing that the presence of QED explanations significantly improves the ability of untrained raters to spot errors made by a strong neural QA baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “Coreference resolution is a capability that many applications require to perform well, including machine translation (Stanovsky et al., 2019; Webster & Pitler, 2020) and question answering (Lamm et al., 2020).”
Teaching language models to support answers with verified quotes (2022, background): “, 2020) that aims to provide natural-language explanations of QA model responses (Lamm et al., 2020; Latcinnik and Berant, 2020; Narang et al., 2020).”
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2020, methodology): “We consider a subset of QED explanations (Lamm et al., 2021), which consist of a sequence of one or more “referential equality annotations” e1 .”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “…open-book science QA 1 fact from W ORLD T REE crowd 5,957 W ORLD T REE V2 (Xie et al., 2020) science exam QA explanation graphs experts 5,100 QED (Lamm et al., 2020) reading comp. QA inference rules authors 8,991 QASC (Khot et al., 2020) science exam QA 2-fact chain authors + crowd 9,980 E QASC…”
A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021, background): “analogous to auto-generated explanations are the ﬁelds of summarisation of text (Tourigny and Capus, 1998; Deutch et al., 2016) and Question-Answering (Dali et al., 2009; Xu et al., 2017; Lamm et al., 2020).”
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning (2022, methodology): “Prior research has also explored using explanations as additional supervision to train joint models (Hancock et al., 2018; Dua et al., 2020; Lamm et al., 2021; Stacey et al., 2022).”
On the Diversity and Limits of Human Explanations (2021, background): “Reference Task Questions and guidelines Lamm et al. (2020) question an- swering referential equality, we cannot infer the exact question Ye et al. (2020) question an- swering Please read carefully to get accepted!”
The Unreliability of Explanations in Few-Shot In-Context Learning (2022, methodology): “Prior research has also explored using explanations as additional supervision to train joint models that jointly predict the labels and explanations (Hancock et al., 2018; Dua et al., 2020; Lamm et al., 2021; Stacey et al., 2022).”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “QED: A framework and dataset for explanations in question707 answering. arXiv preprint arXiv:2009.06354, 2020.”
Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment (2022, methodology): “We thus collect questions for individual passages beforehand similar to Stage 1 and use these as in-148 Commonsense QA Free-form LIAR-PLUS (Alhindi et al., 2018) Fact checking Free-form QED (Lamm et al., 2021) Reading comprehension Structured NExT (Wang et al., 2019) Text classiﬁcation Structured gi”
Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering (2021, background): “Furthermore, our work is complementary to QA explanation efforts like Lamm et al. (2020) that only consider answerable questions.”