A Survey of the State of Explainable AI for Natural Language Processing

By Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, Prithviraj Sen. 2020

470 citations recorded by the source corpus.

Recent years have seen important advances in the quality of state-of-the-art models, but this has come at the expense of models becoming less interpretable. This survey presents an overview of the current state of Explainable AI (XAI), considered within the domain of Natural Language Processing (NLP). We discuss the main categorization of explanations, as well as the various ways explanations can be arrived at and visualized. We detail the operations and explainability techniques currently available for generating explanations for NLP model predictions, to serve as a resource for model developers in the community. Finally, we point out the current gaps and encourage directions for future work in this important research area.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Alignment of Language Agents (2021, background): “For a recent survey of explainable natural language processing (NLP), see Danilevsky et al. (2020).”
Can language models learn from explanations in context? (2022, background): “Note that this question does not focus on explainability as a means to help users understand a model (e.g. Danilevsky et al., 2020); instead, we focus on whether few-shot explanations can help the model itself to “understand” the task, as assessed through performance (cf. Santoro et al., 2021).”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, methodology): “There are many types of techniques to generate feature attribution scores for neural NLP models, as summarized in several recent survey papers on explainability for NLP [39, 132, 135].”
A Review on Language Models as Knowledge Bases (2022, background): “However, current research on LMs is far from achieving this goal, as many of the newer techniques focus on black box rather than white box techniques (Danilevsky et al., 2020).”
Explanation-Based Human Debugging of NLP Models: A Survey (2021, methodology): “…models (Qiu et al., 2020) with many papers studying their behaviors (Rogers et al., 2021; Hoover et al., 2020), only Zylberajch et al. (2021) and Yao et al. (2021) have used pre-trained language models, including BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), as test beds for EBHD.”
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021, background): “The techniques for explainability differ in whether they provide justification or information for model outputs on individual instances (local explainability) or focus on a model as a whole and disclose its internal structure (global explainability) (Danilevsky et al., 2020).”
Causal Reasoning and Large Language Models: Opening a New Frontier for Causality (2023, methodology): “To better understand what aspects of the prompt or question an LLM is attending to, we use redaction and perturbation tests, motivated from explainable AI methods for NLP Sinha et al. (2021); Danilevsky et al. (2020).”
A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts (2021, methodology): “Some of the earlier milestones for the current field of XAI were methods to explain input importance for models with image inputs (Das and Rad, 2020), such as LIME (Ribeiro et al, 2016) and LRP (Bach et al, 2015). Research on interpretability and explainability of models for visual tasks is still ve”
Cross-Domain Transfer of Generative Explanations Using Text-to-Text Models (2021, background): “Explainable ML has received a lot of research interest over the last few years and a comprehensive review of the field in general is provided in [14] and specifically in [8] related to applications for NLP.”
Entailment Tree Explanations via Iterative Retrieval-Generation Reasoner (2022, methodology): “Traditionally, natural language processing (NLP) frameworks were based on white-box methods such as rule-based systems (Allen, 1988; Ribeiro et al., 2019; Ribeiro and Forbus, 2021) and decision trees (Boros et al., 2017), which were inherently in-spectable (Danilevsky et al., 2020).”
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research (2022, methodology): “On the hand, the visualized explanation approaches are used in vaster domains including NLP [65], neural networks [66], and healthcare [67].”
Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities (2021, methodology): “Even though studying the trade-off is essential, it is impossible to proceed without standardized metrics for assessing the quality of explanations [54].”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “The large language model (LLM) interpretability refers to the understanding and explanation of the inner workings and decision-making processes of a large language model [17].”