Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?

By Peter Hase, Shiyue Zhang, Harry Xie, Mohit Bansal. 2020

120 citations recorded by the source corpus.

Data collection for natural language (NL) understanding tasks has increasingly included human explanations alongside data points, allowing past works to introduce models that both perform a task and generate NL explanations for their outputs. Yet to date, model-generated explanations have been evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations. We argue that these evaluations are insufficient, since they fail to indicate whether explanations support actual model behavior (faithfulness), rather than simply match what a human would say (plausibility). In this work, we address the problem of evaluating explanations from the the model simulatability perspective. Our contributions are as follows: (1) We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations, which measures how well explanations help an observer predict a model’s output, while controlling for how explanations can directly leak the output. We use a model as a proxy for a human observer, and validate this choice with two human subject experiments. (2) Using the CoS-E and e-SNLI datasets, we evaluate two existing generative graphical models and two new approaches; one rationalizing method we introduce achieves roughly human-level LAS scores. (3) Lastly, we frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage, which can improve LAS scores.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, background): “This line of research is complementary to works investigating the faithfulness of model-generated free-text explanations (Hase et al., 2020; Wiegreffe et al., 2021).”
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, background): “tions have made no difference to task performance when used as targets (Narang et al., 2020; Hase et al., 2020).”
Measuring Association Between Labels and Free-Text Rationales (2020, background): “Hase et al. (2020) demonstrate that automated metrics for sim-ulatability have moderate to high correlation with human scores in both an expert and a crowdsourced setting.”
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2020, background): “While several quantitative evaluation techniques have recently been proposed, many of these are easily gamed (Treviso and Martins, 2020; Hase et al., 2020).2 Some depend upon the model outputs corresponding to deformed examples that lie outside the support of the training distribution (DeYoung et…”
Post-hoc Interpretability for Neural NLP: A Survey (2021, background): “This sub-field of natural natural language explanations have received criticism in NLP for not evaluating functionally-grounded [48].”
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations (2022, background): “1 for some examples), (3) ensure non-trivial explanations that do not leak the label (Hase et al., 2020), with the aim of avoiding explanations that only repeat the label without providing generalizable background knowledge (a common issue in past explanation datasets), (4) evaluate explanation…”
ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning (2021, methodology): “Following past work on explanation generation (Rajani et al., 2019; Hase et al., 2020), we propose some initial baseline models for our task – (1) a reasoning model that first generates the graph and uses it as additional context with the belief and argument to then predict the stance, and (2) a rat”
Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023, background): “would expect) it to be after reading model explanations (Doshi-Velez and Kim, 2017; Hase et al., 2020; Wiegreffe et al., 2021).”
REV: Information-Theoretic Evaluation of Free-Text Rationales (2022, methodology): “For automatic metrics, the evaluation models of R EV and the proxy models of LAS and RQ are trained on the ECQA training set with gold labels and human-annotated rationales (§3.2).”
Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations (2021, background): “The predictive ability of NLEs over inputs (formally termed as simulatability (DoshiVelez & Kim, 2017; Hase et al., 2020)) is defined by the change in task accuracy when the generated NLEs are appended to the input.”