The Promise and Peril of Human Evaluation for Model Interpretability

By Bernease Herman. 2017

155 citations recorded by the source corpus.

Transparency, user trust, and human comprehension are popular ethical motivations for interpretable machine learning. In support of these goals, researchers evaluate model explanation performance using humans and real world applications. This alone presents a challenge in many areas of artificial intelligence. In this position paper, we propose a distinction between descriptive and persuasive explanations. We discuss reasoning suggesting that functional interpretability may be correlated with cognitive function and user preferences. If this is indeed the case, evaluation and optimization using functional metrics could perpetuate implicit cognitive bias in explanations that threaten transparency. Finally, we propose two potential research directions to disambiguate cognitive function and explanation models, retaining control over the tradeoff between accuracy and interpretability.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

STaR: Bootstrapping Reasoning With Reasoning (2022, background): “While our primary emphasis is not on the explainability benefits that STaR may bring, there is a fundamental challenge around evaluating explanations and rationales: namely, faithfulness [33, 34].”
Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, background): “Herman [18] notes that we should be wary of evaluating interpretable systems using merely human evaluations of interpretability, because human evaluations imply a strong and speciﬁc bias towards simpler descriptions.”
Interpretable Basis Decomposition for Visual Explanation (2018, background): “Since is generally assumed that humans have a strong bias towards simpler explanations ([10]), it is interesting to find that, on average, human raters prefer our longer explanations.”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, background): “fails to distinguish between distinct aspects of the interpretation’s quality, such as readability, plausibility and faithfulness (Herman, 2017).”
RISE: Randomized Input Sampling for Explanation of Black-box Models (2018, background): “As a result, human evaluation has been the predominant way to assess model explanation by measuring it from the perspective of transparency, user trust or human comprehension of the decisions made by the model [11].”
How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation (2018, background): “1As noted in Herman [2017], explanation can also be used to persuade rather than inform; we exclude that use-case here.”
Learning to Faithfully Rationalize by Construction (2020, unknown): “To assess the plausibility of the resulting rationales (Herman, 2017; Wiegreffe and Pinter, 2019), we design a human user study.”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, unknown): “Additionally, besides saving time and resources [162], Herman [96] and Ancona et al.”
Reasoning Models Don't Always Say What They Think (2025, unknown): “Plausibility evaluates the factual correctness of a CoT and how convincingly it justifies the model’s response (Herman, 2017; Lage et al., 2019; Jacovi and Goldberg, 2020).”
Explainability fact sheets: a framework for systematic assessment of explainable approaches (2019, background): “A diﬀerent set of, mostly synthetic ( V2 ), validation approaches was proposed by Herman [15]: • using simulated data with known characteristics to validate correctness of explanations, and • testing stability and consistency of explanations – see the invariance safety requirement ( S3 ) for more de”
Evaluating Explanation Without Ground Truth in Interpretable Machine Learning (2019, background): “computer interaction (HCI) researchers actively explore this area [1, 14], and propose several metrics for human evaluation on general explanation from IML techniques, such as mental mode [20], human-machine performance [9], user satisfaction [19] and user trust [15].”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, background): “In recent years, there have been surveys and position papers suggesting research directions and highlighting challenges in interpretable machine learning research [48, 78, 127].”
A Survey on the Explainability of Supervised Machine Learning (2020, background): “Herman (2017) differentiates between a descriptive and persuasive explanation generation task.”
Faithful Chain-of-Thought Reasoning (2023, background): “3 It should be contrasted with plausibility (a.k.a. persuasive-ness or understandability ), which refers to “how convincing an explanation is to humans” (Herman, 2019; Jacovi and Goldberg, 2020).”
Machine Learning Testing: Survey, Landscapes and Horizons (2019, background): “Some work has presented definitions or statistical measurements of non-functional features of ML systems including robustness [137], fairness [138], [139], [140], and interpretability [65], [141].”
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020, background): “Their limitation is that they can be used to persuade users that the model is reliable when it is not (Bansal et al., 2020)—an ethical issue raised by Herman (2017).”
Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning (2018, background): “Herman [18] notes that we should be wary of evaluating interpretable systems using merely human evaluations of interpretability, because human evaluations imply a strong and speciﬁc bias towards simpler descriptions.”
Aligning Faithful Interpretations with their Social Attribution (2020, background): “Plausibility (Wiegreffe and Pinter, 2019) or persuasiveness (Herman, 2017) is the property of how”