AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models

By Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, Sameer Singh. 2019

146 citations recorded by the source corpus.

Neural NLP models are increasingly accurate but are imperfect and opaque—they break in counterintuitive ways and leave end users puzzled at their behavior. Model interpretation methods ameliorate this opacity by providing explanations for specific model predictions. Unfortunately, existing interpretation codebases make it difficult to apply these methods to new models and tasks, which hinders adoption for practitioners and burdens interpretability researchers. We introduce AllenNLP Interpret, a flexible framework for interpreting NLP models. The toolkit provides interpretation primitives (e.g., input gradients) for any AllenNLP model and task, a suite of built-in interpretation methods, and a library of front-end visualization components. We demonstrate the toolkit’s flexibility and utility by implementing live demos for five interpretation methods (e.g., saliency maps and adversarial attacks) on a variety of models and tasks (e.g., masked language modeling using BERT and reading comprehension using BiDAF). These demos, alongside our code and tutorials, are available at https://allennlp.org/interpret.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, background): “Choices for f include L 1 or L 2 norm (Atanasova et al., 2020), or an element-wise sum (Wallace et al., 2019).”
ExplainaBoard: An Explainable Leaderboard for NLP (2021, background): “On the other hand, there are many other glass-box interpretability tools that look at model internals, such as the AllenNLP Interpret (Wallace et al., 2019) and Language Interpretability Tool (Tenney et al.”
TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP (2020, background): “Most recently, AllenNLP Interpret includes functionality for running adversarial attacks on NLP models, but is intended only for the purpose of interpretability, and only supports attacks via input-reduction or greedy gradient-based word swap (Wallace et al., 2019).”
What the DAAM: Interpreting Stable Diffusion Using Cross Attention (2022, background): “A large body of relevant papers exists, where researchers apply textual perturbation (Wallace et al., 2019), attention visualization (Vig, 2019; Kovaleva et al., 2019; Shimaoka et al., 2016), and information bottlenecks (Jiang et al., 2020) to relate important input tokens to the outputs of large…”
RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms (2020, methodology): “We use the SmoothGrad (Smilkov et al., 2017) algorithm from AllenNLP Interpret (Wallace et al., 2019) for masked word prediction on our probes with real people’s names (the same set as our ablation study) using BERT.”
The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models (2020, methodology): “…process, but many come with limitations: ofﬂine tools such as TFMA (Mewald, 2019) can provide only aggregate metrics, interactive frontends (e.g. Wallace et al., 2019) may focus on single-datapoint explanation, and more integrated tools (e.g. Wexler et al., 2020; Mothilal et al., 2020;…”
Interpreting Predictions of NLP Models (2020, methodology): “Alongside the tutorial, we will present source code implementations of various interpretation methods using AllenNLP Interpret (Wallace et al., 2019b).”
Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluation (2022, background): “Wallace et al. (2019) presented an interpretation framework where they improved a visual component of the model prediction process on several NLP tasks for the end-user.”
Robustness Gym: Unifying the NLP Evaluation Landscape (2021, background): “, 2019), AllenNLP Interpret (Wallace et al., 2019b), InterpretML (Nori et al.”