Explaining and Interpreting LSTMs

By Leila Arras, José Antonio Arjona-Medina, Michael Widrich, Grgoire Montavon, Michael Gillhofer, K. Müller, Sepp Hochreiter, Wojciech Samek. 2019

89 citations recorded by the source corpus.

While neural networks have acted as a strong unifying force in the design of modern AI systems, the neural network architectures themselves remain highly heterogeneous due to the variety of tasks to be solved. In this chapter, we explore how to adapt the Layer-wise Relevance Propagation (LRP) technique used for explaining the predictions of feed-forward networks to the LSTM architecture used for sequential data modeling and forecasting. The special accumulators and gated interactions present in the LSTM require both a new propagation scheme and an extension of the underlying theoretical framework to deliver faithful explanations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Explainable AI Methods - A Brief Overview (2020, background): “For instance, Arras et al. [7, 6] developed meaningful LRP redistribution rules for LSTM models.”
Modern Hopfield Networks and Attention for Immune Repertoire Classification (2020, methodology): “, 2017) or Layer-Wise Relevance Propagation (Montavon et al., 2018; Arras et al., 2019).”
Toward Interpretable Machine Learning: Transparent Deep Neural Networks and Beyond (2020, methodology): “Additional propagation rules have been proposed for special layers such as min/max pooling [13], [99], [68] and LSTM blocks [11], [9].”
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (2019, methodology): “Layer-wise Relevance Propagation (LRP) [9,58] is a propagation-based explanation framework, which is applicable to general neural network structures, including deep neural networks [13], LSTMs [5,7], and Fisher Vector classifiers [44].”
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications (2021, methodology): “Other neural network architectures have a more conventional structure but still require a nontrivial adaptation of existing explanation methods, for example, extensions of LRP have been proposed to deal with the special LSTM blocks in recurrent neural networks [9], [11] or to handle attention units ”
Explainable artificial intelligence: a comprehensive review (2021, methodology): “For the first trend, Arras et al. (2019) adapted the LRP algorithm that was usually used in the feed-forward networks to interpret the RNN predictions.”