Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?

By Alon Jacovi, Yoav Goldberg. 2020

897 citations recorded by the source corpus.

With the growing popularity of deep-learning based NLP models, comes a need for interpretable systems. But what is interpretability, and what constitutes a high-quality interpretation? In this opinion piece we reflect on the current state of interpretability evaluation research. We call for more clearly differentiating between different desired criteria an interpretation should satisfy, and focus on the faithfulness criteria. We survey the literature with respect to faithfulness evaluation, and arrange the current approaches around three assumptions, providing an explicit form to how faithfulness is “defined” by the community. We provide concrete guidelines on how evaluation of interpretation methods should and should not be conducted. Finally, we claim that the current binary definition for faithfulness sets a potentially unrealistic bar for being considered faithful. We call for discarding the binary notion of faithfulness in favor of a more graded one, which we believe will be of greater practical utility.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, background): “An explanation is faithful only if it reflects the information that is used for decision making (Lipton, 2018; Jacovi and Goldberg, 2020), which these systems do not guarantee.”
SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, background): “A good model explanation should be (i) relevant to the current input and predictions and (ii) understandable to humans (DeYoung et al., 2020; Jacovi and Goldberg, 2020; Wiegreffe et al., 2020; Jain et al., 2020).”
Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Jacovi and Goldberg (2020) argue for the development of evaluations that measure the extent and likelihood that a rationale is faithful in practice (illustrated in Figure 1).”
Teaching language models to support answers with verified quotes (2022, methodology): “However, while our training objective incentivises the model’s answer to agree with the evidence it provides, our method makes no attempt to guarantee that the evidence faithfully (Jacovi and Goldberg, 2020) describes the reason that the model generated the claim.”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, background): “This can both ensure consistency with external sources of truth, and improve the model’s faithfulness via connecting the verdict with a speciﬁc evidence (Jacovi and Goldberg, 2020; Ross et al., 2020).”
How Do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020, background): “, 2020), will not help us in demonstrating this, as humans cannot judge if an interpretation is faithful (Jacovi and Goldberg, 2020).”
QED: A Framework and Dataset for Explanations in Question Answering (2020, background): “The explanations provided in QED are discrete structured objects, as opposed, for example, to “heat map"-style explanations (attention distributions, or other real-valued, wordlevel feature importance measures) (Jacovi and Goldberg, 2020).”
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI (2020, background): “The recent rapid growth in the field of explainable AI (XAI) is also, in part, motivated by the need to maintain trust between the human user and the AI [31, 35, 46, 52, 59, 72].”
Think about it! Improving defeasible reasoning by first modeling the question scenario. (2021, background): “Jacovi and Goldberg (2020) note that an explanation should aim towards two complementary goals: i) Plausibility: provide an interpretation of system outputs that is convincing for humans, and ii) Faithfulness: capture the actual reasoning process of a model.”
REV: Information-Theoretic Evaluation of Free-Text Rationales (2022, background): “Faithfulness measures to which extent rationales reflect the true reasoning process of models, while plausibility evaluates how convincing rationales are to humans (Jacovi and Goldberg, 2020).”
Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking (2020, methodology): “Due to the complexity, no human gold standard for attribution can be constructed in this setting (Jacovi & Goldberg, 2020).”