Making deep neural networks right for the right scientific reasons by interacting with their explanations

By Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, Kristian Kersting. 2020

250 citations recorded by the source corpus.

Deep neural networks have demonstrated excellent performances in many real-world applications. Unfortunately, they may show Clever Hans-like behaviour (making use of confounding factors within datasets) to achieve high performance. In this work we introduce the novel learning setting of explanatory interactive learning and illustrate its benefits on a plant phenotyping research task. Explanatory interactive learning adds the scientist into the training loop, who interactively revises the original model by providing feedback on its explanations. Our experimental results demonstrate that explanatory interactive learning can help to avoid Clever Hans moments in machine learning and encourages (or discourages, if appropriate) trust in the underlying model. Deep learning approaches can show excellent performance but still have limited practical use if they learn to predict based on confounding factors in a dataset, for instance text labels in the corner of images. By using an explanatory interactive learning approach, with a human expert in the loop during training, it becomes possible to avoid predictions based on confounding factors.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Can language models learn from explanations in context? (2022, methodology): “For example, explanations have been used for training or tuning models for computer vision (Hendricks et al., 2016; Schramowski et al., 2020; Majumder et al., 2021; Mu et al., 2020), for mathematical problem solving by program induction (Ling et al., 2017), or relational reasoning and reinforcement…”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Black box models often predict the right answer for the wrong reason (the “Clever Hans” phenomenon), leading to poor performance in practice (Schramowski et al., 2020; Lapuschkin et al., 2019).”
A case-based interpretable deep learning model for classification of mass lesions in digital mammography (2021, background): “Ways to direct model attention include data augmentation [16, 33, 8], techniques that combat hand-selected confounders [51, 50], techniques that combat learned confounders [58], and an approach where a human critic manually approves the attention map during training [43].”
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, methodology): “In fact, the use of human visual explanation has been investigated by recent works in several supervised learning tasks [39, 35, 34].”
Tell me why! - Explanations support learning of relational and causal structure (2021, background): “Thus, there has been increasing interest in using explanations as learning signal (e.g. Ross et al., 2017; Mu et al., 2020; Camburu et al., 2018; Schramowski et al., 2020; see related work).”
Large pre-trained language models contain human-like biases of what is right and wrong to do (2021, background): “Here, explanatory interactive learning [47, 48, 49] is promising as it includes a mechanism enabling the AI system to explain its’ choices as well as a revision based on these explanations.”
Improving performance of deep learning models with axiomatic attribution priors and expected gradients (2020, methodology): “To choose λ , we searched over values in [10 − 20 , 10 − 1 ] and chose the λ that minimized the attribution prior penalty and achieved a test accuracy within 1% of the baseline model for MNIST and 10% for CIFAR-10.”
Tell me why! Explanations support learning relational and causal structure (2021, background): “Indeed, there has been increasing interest in using language or explanations as a learning signal for machines (e.g., Ross et al., 2017; Mu et al., 2020; Camburu et al., 2018; Schramowski et al., 2020; see related work).”
Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations (2020, methodology): “ToyColor [43] 40k 2 5× 5× 3 ✗ ✓ ✗ 1 ColorMNIST [17] 70k 10 28× 28× 3 ✗ ✗ ✓ 1 Decoy-MNIST [43] 70k 10 28× 28× 3 ✗ ✓ ✗ 1 Plant Data Set [44] 2.”
Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations (2021, background): “Moreover, self-explanatory models may beneﬁt from the rich information in the explanations provided at training time (Schramowski et al., 2020; Stacey et al., 2022; Lazaridou et al., 2022).”