Interpretable Neural Predictions with Differentiable Binary Variables

By Jasmijn Bastings, Wilker Aziz, Ivan Titov. 2019

236 citations recorded by the source corpus.

The success of neural networks comes hand in hand with a desire for more interpretability. We focus on text classifiers and make them more interpretable by having them provide a justification–a rationale–for their predictions. We approach this problem by jointly training two neural network models: a latent model that selects a rationale (i.e. a short and informative part of the input text), and a classifier that learns from the words in the rationale alone. Previous work proposed to assign binary latent masks to input positions and to promote short selections via sparsity-inducing penalties such as L0 regularisation. We propose a latent model that mixes discrete and continuous behaviour allowing at the same time for binary selections and gradient-based training without REINFORCE. In our formulation, we can tractably compute the expected value of penalties such as L0, which allows us to directly optimise the model towards a pre-specified text selection rate. We show that our approach is competitive with previous work on rationale extraction, and explore further uses in attention mechanisms.

Subfields: Artificial Intelligence, Model Architectures, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Safety, Robustness & Interpretability

Citations in the corpus

Few-Shot Self-Rationalization with Natural Language Prompts (2021, methodology): “Since then, many highlighting methods have been designed following their approach (Bastings et al., 2019; Yu et al., 2019; Paranjape et al., 2020).”
SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, methodology): “(2020), we use the same experimental setup and saliency-based baselines such as attention (Lei et al., 2016; Bastings et al., 2019) and gradient (Li et al.”
EDUCE: Explaining model Decisions through Unsupervised Concepts Extraction (2019, methodology): “Such methods are speciﬁc to text data select and rationales, i.e. parts of text, on which the model bases its prediction, see [15, 27] and very recently [5].”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “Methods that learn to explain by extracting rationales from the input and only using those rationales to make a prediction are faithful by design [8, 13, 41, 60].”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, methodology): “Finally, other recent work has proposed using a differentiable binary mask over inputs, which also avoids recourse to REINFORCE (Bastings et al., 2019).”
How Do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020, methodology): “To the same end, Bastings et al. (2019) minimize downstream loss subject to constraints on expected L 0 using a variant of the sparse relaxation of Louizos et al. (2018).”
Why Attention is Not Explanation: Surgical Intervention and Causal Reasoning about Neural Models (2020, background): “Additionally, in human-computer interaction researchers have worked to determine what users want from explanations (Lim and Dey, 2009), for instance by showing uses only a subset of text highlighted as important (as a simplifying step) (Bastings et al., 2019).”
Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking (2020, methodology): “To enable gradient-based optimization for our erasure function, we rely on sparse stochastic gates (Louizos et al., 2018; Bastings et al., 2019).”
Cross-Domain Transfer of Generative Explanations Using Text-to-Text Models (2021, background): “Another line of work for explainable NLP is based on rationalization pipelines that aim to produce extractive explanations by splitting the prediction process into two subsequent modules; a rationale extractor and a predictor [1,10,25,34].”
Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? (2020, background): “Past works on model interpretability have observed trade-offs between accuracy and model constraints for interpretation purposes (Bastings et al., 2019; Jain et al., 2020).”
Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases (2021, methodology): “To address this problem, Lei et al. (2016) used reinforcement learning (Williams, 1992), while Bastings et al. (2019) proposed a differentiable mechanism relying on the re-parameterization trick (Louizos and Welling, 2017).”
Rationale-Inspired Natural Language Explanations with Commonsense (2021, methodology): “timent in reviews) and later was successfully followed by works for both NL (DeYoung et al., 2020; Lei et al., 2016; Bastings et al., 2019; Sha et al., 2021) and VL (Strout et al.”
Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations (2021, methodology): “We follow Bastings et al. (2019) and optimize min θ r ,θ g ,θ ks ,θ p L 1 with where the second term is the L 1 penalty, the third term is a fused Lasso to control the total number of transitions for compactness (Lei et al., 2016), and λ r 0 and λ r 1 are hyperpa-rameters.”
Explaining and Improving Model Behavior with k Nearest Neighbor Representations (2020, methodology): “…for NLP models have focused on explaining individual predictions by using gradient-based saliency maps over the input text (Lei, Barzilay, and Jaakkola 2016; Ribeiro, Singh, and Guestrin 2018; Bastings, Aziz, and Titov 2019) or interpreting attention (Brunner et al. 2020; Pruthi et al. 2020).”
Rethinking Cooperative Rationalization: Introspective Extraction and Complement Control (2019, background): “Another related concurrent work (Bastings et al., 2019) proposes differentiable solution to optimize the cooperative rationalization method.”
Structured Pruning of Large Language Models (2019, methodology): “We prune the mask during training via l 0 regularization, and use an augmented Lagrangian approach inspired by (Bastings et al., 2019) to control the ﬁnal sparsity level of the model.”