Why Interpretability in Machine Learning? An Answer Using Distributed Detection and Data Fusion Theory
By Kush R. Varshney, Prashant Khanduri, Pranay Sharma, Shan Zhang, Pramod K. Varshney. 2018
24 citations recorded by the source corpus.
As artificial intelligence is increasingly affecting all parts of society and life, there is growing recognition that human interpretability of machine learning models is important. It is often argued that accuracy or other similar generalization performance metrics must be sacrificed in order to gain interpretability. Such arguments, however, fail to acknowledge that the overall decision-making system is composed of two entities: the learned model and a human who fuses together model outputs with his or her own information. As such, the relevant performance criteria should be for the entire system, not just for the machine learning component. In this work, we characterize the performance of such two-node tandem data fusion systems using the theory of distributed detection. In doing so, we work in the population setting and model interpretable learned models as multi-level quantizers. We prove that under our abstraction, the overall system of a human with an interpretable classifier outperforms one with a black box classifier.
Subfields: Artificial Intelligence, Computer Networks and Communications
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tongshuang Sherry Wu
- Daniel S. Weld
- Marco Tulio Ribeiro
- Ece Kamar
- D. V. Carvalho
- Eduardo Marques Pereira
- Jaime S. Cardoso
- Gagan Bansal
In Computer Networks and Communications
Citations in the corpus
- Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, background): “[169] tend to base their proposal on a detailed problem formulation that becomes invalid as soon as the method of interpretability or the explained model change.”
- Machine Learning Interpretability: A Survey on Methods and Metrics (2019, background): “Thus, for the success of AI as an indispensable tool for decision support, it must not only bring forth relevant information for the decision making task but also communicate at an appropriate rate and in a way that allows a human recipient of the information to tap into the strengths of intuition a”
- Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, background): “As recent work has suggested [4, 55, 59, 79], it may be useful to directly optimize for complementary behavior by accounting for the human behavior during training, who may have access to a diferent set of features [72].”