A causal framework for explaining the predictions of black-box sequence-to-sequence models

By David Alvarez-Melis, Tommi S. Jaakkola. 2017

213 citations recorded by the source corpus.

We interpret the predictions of any black-box structured input-structured output model around a specific input-output pair. Our method returns an “explanation” consisting of groups of input-output tokens that are causally related. These dependencies are inferred by querying the model with perturbed inputs, generating a graph over tokens from the responses, and solving a partitioning problem to select the most relevant components. We focus the general approach on sequence-to-sequence problems, adopting a variational autoencoder to yield meaningful input perturbations. We test our method across several NLP sequence generation tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

e-SNLI: Natural Language Inference with Natural Language Explanations (2018, background): “Alvarez-Melis and Jaakkola [1] introduces a similar approach but for structured prediction, where a variational autoencoder provides relevant perturbations of the inputs that are then used to infer pairs of input-output tokens that are causally related.”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “Alvarez-Melis and Jaakkola (2017) discovered input-output associations in a sequence-to-sequence learning scenario, by perturbing the input and ﬁnding the most relevant associations.”
Attention is not Explanation (2019, background): “Another novel line of work in this direction involves explicitly identifying explanations of black-box predictions via a causal framework (Alvarez-Melis and Jaakkola, 2017).”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “In text classification, [126] provides explanations in the form”
EDUCE: Explaining model Decisions through Unsupervised Concepts Extraction (2019, background): “[3] design a model that detects input-output pairs that are causally related.”
Do Not Trust Additive Explanations (2019, methodology): “, 2017), text data (Marco Tulio Ribeiro and Sameer Singh and Carlos Guestrin, 2018; Jacovi et al., 2018; Alvarez-Melis and Jaakkola, 2017), and tabular data (Molnar, 2019; Biecek, 2018).”
Causal Interpretability for Machine Learning - Problems, Methods and Evaluation (2020, methodology): “Alvarez-Melis and Jaakkola [3] generated causal explanations for structured input structured output black-box models by (a) generating perturbed samples using a variational auroencoder; (b) generating a weighted bipartite graph G = (Vx ∪ Vy, E), where Vx and Vy are elements in x and y and Eij repres”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “Examples include approaches that fit a linear classifier to a model’s prediction on perturbations of the input [4, 59, 70].”
Towards Robust Interpretability with Self-Explaining Neural Networks (2018, background): “The interpretations are derived locally, around each example, on the basis of limited access to the inner workings of the model such as gradients or reverse propagation [4, 18], or through oracle queries to estimate simpler models that capture the local input-output behavior [16, 2, 14].”
Generating Natural Adversarial Examples (2017, background): “Prior work on generating textual adversaries [1, 12, 17] performs word erasures and replacements directly on text input space x, using domain-specific rule based or heuristic based approaches, or requires manual intervention.”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, background): “, 2016) and Alvarez-Melis and Jaakkola (2017); these methods approximate model behavior locally by having it repeatedly make predictions over perturbed inputs and fitting a simple, explainable model over the outputs.”
Interpretable Neural Predictions with Differentiable Binary Variables (2019, methodology): “Ribeiro et al. (2016) make any classiﬁer interpretable by approximating it locally with a linear proxy model in an approach called LIME, and Alvarez-Melis and Jaakkola (2017) propose a framework that returns input-output pairs that are causally related.”
A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions (2019, background): “In general, making concrete statements about the causal link between inputs and outputs of sequence-to-sequence models is challenging, even in the text-to-text case, see Alvarez-Melis and Jaakkola (2017).”
On the Robustness of Interpretability Methods (2018, methodology): “On the other hand, perturbation-based methods rely on querying the model around the prediction of interest to infer relevance of input features towards the output (Ribeiro et al., 2016; Alvarez-Melis & Jaakkola, 2017).”
Evaluating Explanation Methods for Neural Machine Translation (2020, methodology): “els (Ding et al., 2017; Alvarez-Melis and Jaakkola, 2017; Ding et al., 2019; He et al., 2019).”
Latent Alignment and Variational Attention (2018, background): “However, more so than these other components, which are often treated as black-boxes, researchers use intermediate attention decisions directly as a tool for model interpretability [43, 1] or as a factor in ﬁnal predictions [25, 68].”
EXS: Explainable Search Using Local Model Agnostic Interpretability (2018, methodology): “Local model agnostic interpretability techniques [1, 13, 14] produce explanations for a single decision by linearly approximating the local behavior of a complex blackbox model. approximation is attained by training a simple linear model in a new interpretable feature space. training data for the…”
A Game Theoretic Approach to Class-wise Selective Rationalization (2019, background): “Finally, the third class is locally ﬁtting a deep network with interpretable models, such as linear models [2, 24].”