Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems

By Richard J. Tomsett, Dave Braines, Daniel Harborne, Alun David Preece, Supriyo Chakraborty. 2018

191 citations recorded by the source corpus.

Several researchers have argued that a machine learning system’s interpretability should be defined in relation to a specific agent or task: we should not ask if the system is interpretable, but to whom is it interpretable. We describe a model intended to help answer this question, by identifying different roles that agents can fulfill in relation to the machine learning system. We illustrate the use of our model in a variety of scenarios, exploring how an agent’s role influences its goals, and the implications for defining interpretability. Finally, we make suggestions for how our model could be useful to interpretability researchers, system developers, and regulatory bodies auditing machine learning systems.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, methodology): “Adapted from H€ ullermeier and Waegeman,(15) Lee and See,(16) Nilsson,(17) and Tomsett et al.(18) ll OPEN ACCESS Perspective”
Manipulating and Measuring Model Interpretability (2018, background): “This problem is exacerbated by the fact that there are different types of users of machine learning systems and these users may have different needs in different scenarios [25].”
Explaining machine learning classifiers through diverse counterfactual explanations (2019, background): “Besides helping to debug ML models, explanations are hypothesized to improve the interpretability and trustworthiness of algorithmic decisions and enhance human decision making [13, 23, 25, 38].”
Stakeholders in Explainable AI (2018, background): “In our own recent work, we examined explainability and interpretability from the perspective of explanation recipients, of six kinds (Tomsett et al. 2018): system creators, system operators, executors making a decision on the basis of system outputs, decision subjects affected by an executor’s decis”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, background): “Instead, it should be dened by user behavior, that is, whether model explanations can improve people’s abilities in completing various tasks [24, 59], and the “people” here can be dierent parties in the AI ecosystems including model developers, regulators, and end-users [63, 64].”
Techniques for interpretable machine learning (2018, background): “tions Explanations might be delivered as a conversation between the explainer and explanation receivers. It means that we need to consider the social context, i.e., to whom an explanation is provided [35], in order to determine the content and formats of explanations. For instance, a preferred forma”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, background): “, [38, 200]) also provide useful compilations of terminology and concepts in comprehensive reports.”
Explainable AI for Intelligence Augmentation in Multi-Domain Operations (2019, background): “In recent work, we examined explainability from the perspective of explanation recipients, of six kinds (Tomsett et al. 2018): system creators , system operators , executors making a decision on the basis of system outputs, decision subjects affected by an executor’s decision, data subjects whose…”
Trust in AI: progress, challenges, and future directions (2024, unknown): “…that the level of interpretability of a system should depend on the category of the entity working with the system, e.g., operators, executors, examiners, and the system should reveal a set of suitable measures of interpretability based on their relation to the system (Tomsett et al., 2018).”
Explainability fact sheets: a framework for systematic assessment of explainable approaches (2019, background): “Therefore, discussing the level and type of background knowledge required to comprehend an explanation is crucial [4, 41, 52].”
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI (2021, methodology): “Because they may be accountable for their decisions, they utilize explanations to assure the underlying models is trustworthy (i.e., "they can reasonably trust a model’s outputs" [5]) ( operator-interpretability [56]).”
Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning (2020, result): “Our results highlight challenges for designing interpretability tools for data scientists, and, in line with prior work [49, 62], we advocate for similar user-centric evaluations to be conducted for all stakeholders of interpretability tools and ML models.”
Solving the Black Box Problem: A Normative Framework for Explainable Artificial Intelligence (2019, background): “Finally, examiners are “ agents tasked with compliance/safety-testing, auditing, or forensically investigating a system ” (Tomsett et al. 2018, p. 13).”