Unmasking Clever Hans predictors and assessing what machines really learn

By Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grgoire Montavon, Wojciech Samek, K. Müller. 2019

1,223 citations recorded by the source corpus.

Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly intelligent behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner. Nonlinear machine learning methods have good predictive ability but the lack of transparency of the algorithms can limit their use. Here the authors investigate how these methods approach learning in order to assess the dependability of their decision making.

Subfields: Artificial Intelligence, Management Science and Operations Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Safety, Robustness & Interpretability

Citations in the corpus

Natural Adversarial Examples (2019, background): “Models may learn spurious cues and obtain high accuracy, but for the wrong reasons [38, 16].”
CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, methodology): “Our contributions are: (a) we propose CLOOB, a new contrastive learning method, (b) we propose modern Hopﬁeld networks to reinforce covariance structures, (c) we propose InfoLOOB as an objective to avoid saturation as observed with InfoNCE, and provide theoretical justiﬁcations for optimizing…”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Black box models often predict the right answer for the wrong reason (the “Clever Hans” phenomenon), leading to poor performance in practice (Schramowski et al., 2020; Lapuschkin et al., 2019).”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “In particular, [12] shows how visualization is capable of helping researchers detect erroneous reasoning in classification problems that many previous researchers have missed, In this work, we survey through research works related to the interpretability of ML or computer algorithms in general,…”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “In fact, a prominent recent work on “Unmasking Clever Hans” predictors (Lapuschkin et al., 2019) has shown that the decision making of a discriminative deep neural network can be based on entirely trivial features, such as a certain object always occurring at a speciﬁc location in every image or…”
Fairwashing Explanations with Off-Manifold Detergent (2020, background): “The interest in explainability is partly driven by the hope that explanations can act as proof for a sensible, fair, and trustworthy decision-making process(Aı̈vodji et al., 2019; Lapuschkin et al., 2019).”
A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts (2021, background): “An example for getting from local to global explanations SpRAy [83] is SpRAy [83] (Spectral Relevance Analysis).”
Benchmarking and survey of explanation methods for black box models (2021, background): “A variant of -lrp is spray [80] which builds a specrtal clustering on top of the local instance-based -lrp explanations.”
Analyzing ImageNet with Spectral Relevance Analysis: Towards ImageNet un-Hans'ed (2019, methodology): “Following [7], we provide attributions computed with Layer-wise Rel-evance Propagation (LRP) [10] according to the recommended composite (or layer-dependent) strategy [23, 24].”