Increasing the Interpretability of Recurrent Neural Networks Using Hidden Markov Models

By Viktoriya Krakovna, Finale Doshi-Velez. 2016

71 citations recorded by the source corpus.

As deep neural networks continue to revolutionize various application domains, there is increasing interest in making these powerful models more understandable and interpretable, and narrowing down the causes of good and bad predictions. We focus on recurrent neural networks (RNNs), state of the art models in speech recognition and translation. Our approach to increasing interpretability is by combining an RNN with a hidden Markov model (HMM), a simpler and more transparent model. We explore various combinations of RNNs and HMMs: an HMM trained on LSTM states; a hybrid model where an HMM is trained first, then a small LSTM is given HMM state distributions and trained to fill in gaps in the HMM's performance; and a jointly trained hybrid model. We find that the LSTM and HMM learn complementary information about the features in the text.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “Finally, [284] constructs a combination of an HMM (Hidden Markov Model) and an RNN, so that the overall model approach harnesses the interpretability of the HMM and the accuracy of the RNN model.”
Interpretable Recurrent Neural Networks Using Sequential Sparse Recovery (2016, neutral): “Krakovna and Doshi-Velez [6] proposed increasing the interpretability of RNNs by combining hidden Markov models with LSTMs. Unlike this work, our goal is not human interpretability, but model interpretability, which means our proposed SISTA-RNN uses no black box components like LSTMs that are not…”
Adversarial Risk and the Dangers of Evaluating Against Weak Attacks (2018, background): “parency is often measured qualitatively from the perspective of a human (Kim, 2015; Krakovna and Doshi-Velez, 2016; Ross and Doshi-Velez, 2018), we note that obscurity as we define it provides one quantitative measure of the degree to which models can be easily analyzed by other algorithms.”
How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation (2018, methodology): “…[Caruana et al., 2015, Kim et al., 2015a, R¨uping, 2006, Bucilu et al., 2006, Ustun and Rudin, 2016, Doshi-Velez et al., 2015, Kim et al., 2015b, Krakovna and Doshi-Velez, 2016, Hughes et al., 2016, Jung et al., 2017], various kinds of logic-based methods [Wang and Rudin, 2015, Lakkaraju et…”
Human-in-the-loop machine learning: a state of the art (2022, background): “…understanding what an RNN model has learned, mainly via feature relevance methods (Arras et al. 2017; Che et al. 2015); and (2) explainability by modifying RNN architectures to provide insights about the decisions they make via local explanations (Krakovna and Doshi-Velez 2016; Choi et al. 2016).”
Interpreting Word-Level Hidden State Behaviour of Character-Level LSTM Language Models (2018, methodology): “This also bears some similarities to a visualization technique used by Krakovna and Doshi-Velez (2016) to investigate a hybrid HMM-LSTM model, although their work uses only 10 k-means clusters and does not deeply investigate clustering.”
Explainable Product Search with a Dynamic Relation Embedding Model (2019, background): “While there have been a variety of studies on the transparency of complicated systems, such as the interpretability of a machine learning model [27, 31, 42] and the importance of different ranking features [25, 58], the concept of explainability in this paper refers to a system’s ability on…”
An Evaluation of the Human-Interpretability of Explanation (2019, methodology): “General Human Factors for Interpretable ML Closer to the objectives of this work, Kulesza et al. [2013] performed a qualitative study in which they varied the soundness (nothing but the truth) and the completeness (the whole truth) of an explanation in a recommendation system setting.”