Learning to Faithfully Rationalize by Construction

By Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, Byron C. Wallace. 2020

177 citations recorded by the source corpus.

In many settings it is important for one to be able to understand why a model made a particular prediction. In NLP this often entails extracting snippets of an input text ‘responsible for’ corresponding model output; when such a snippet comprises tokens that indeed informed the model’s prediction, it is a faithful explanation. In some settings, faithfulness may be critical to ensure transparency. Lei et al. (2016) proposed a model to produce faithful rationales for neural text classification by defining independent snippet extraction and prediction modules. However, the discrete selection over input tokens performed by this method complicates training, leading to high variance and requiring careful hyperparameter tuning. We propose a simpler variant of this approach that provides faithful explanations by construction. In our scheme, named FRESH, arbitrary feature importance scores (e.g., gradients from a trained model) are used to induce binary labels over token inputs, which an extractor can be trained to predict. An independent classifier module is then trained exclusively on snippets provided by the extractor; these snippets thus constitute faithful explanations, even if the classifier is arbitrarily complex. In both automatic and manual evaluations we find that variants of this simple framework yield predictive performance superior to ‘end-to-end’ approaches, while being more general and easier to train. Code is available at https://github.com/successar/FRESH.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, methodology): “By design, we ensure that the generated proof is solely used for determining the label of claim, making it a faithful by construction model (Lei et al., 2016; Jain et al., 2020).”
SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, methodology): ““Faithfulness-by-construction” (FRESH) pipeline (Jain et al., 2020) is an example of such framework to evaluate sufficiency of explanations: the sole explanations, without the remaining parts of the input, must be sufficient for predicting a label.”
Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Such models approach faithfulness by construction, if there is no gradient flow between the explainer and predictor modules (Jain et al., 2020).”
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions (2020, methodology): “2 Here we focus on interpretability approaches which are faithful (Wiegreffe and Pinter, 2019; Jacovi and Goldberg, 2020; Jain et al., 2020) by construction; other approaches are discussed in §6.”
Aligning Faithful Interpretations with their Social Attribution (2020, methodology): “Citing significant difficulty in training these solutions, Jain et al. (2020) have proposed to train the selector and predictor separately, by first training a separate model on the end task, and using an unfaithful highlight interpretation method on this model to select a highlight, then training…”
An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction (2020, background): “, 2017; AlvarezMelis and Jaakkola, 2017) and analyses of selfattention (Serrano and Smith, 2019; Jain et al., 2020).”
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020, background): “, 2020) or a pipeline model (Camburu et al., 2018; Rajani et al., 2019; Jain et al., 2020) for producing both predictions and natural language rationales.”
Detect and Perturb: Neutral Rewriting of Biased and Sensitive Text via Gradient-based Decoding (2021, unknown): “Following Jain et al. (2020), we take self-attention scores of all input tokens w.r.t. the [CLS] token (Devlin et al., 2018) from the ﬁnal hidden layers and normalize them to measure how salient each token is for predicting A .”
Explaining NLP Models via Minimal Contrastive Editing (MiCE) (2020, background): “diction (Zaidan et al., 2007; Lei et al., 2016; Chang et al., 2019; Bastings et al., 2019; Yu et al., 2019; DeYoung et al., 2020; Jain et al., 2020; Belinkov and Glass, 2019) that can be derived through gradients (Simonyan et al.”
Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision (2020, methodology): “…growing interest in NLP in opening the black-box, through surrogate models (Ribeiro, Singh, and Guestrin 2016), counterfactual evaluation (Ten-ney et al. 2020), examining the inner structure of the neural network (Raffel et al. 2017; Jain et al. 2020), or generating natural language explanations.”
REV: Information-Theoretic Evaluation of Free-Text Rationales (2022, background): “Ex-tractive rationales contain some important features extracted from input texts that make models produce final predictions (Lei et al., 2016; DeYoung et al., 2020; Jain et al., 2020; Schulz et al., 2020).”
Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? (2020, background): “Past works on model interpretability have observed trade-offs between accuracy and model constraints for interpretation purposes (Bastings et al., 2019; Jain et al., 2020).”
Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases (2021, methodology): “2, has comparable performance and better rationale quality, compared to a model with two independent paragraph encoders, as the one used in the literature (Lei et al., 2016; Yu et al., 2019; Jain et al., 2020).”
Multi-document Summarization via Deep Learning Techniques: A Survey (2020, background): “The idea of using adversarial examples to examine the robustness of a DNN model originated from research in Computer Vision [144] and was introduced in NLP by Jia et al. [70].”