Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees
By Guiliang Liu, Oliver Schulte, Wang Bill Zhu, Qingcan Li. 2018
157 citations recorded by the source corpus.
Deep Reinforcement Learning (DRL) has achieved impressive success in many applications. A key component of many DRL models is a neural network representing a Q function, to estimate the expected cumulative reward following a state-action pair. The Q function neural network contains a lot of implicit knowledge about the RL problems, but often remains unexamined and uninterpreted. To our knowledge, this work develops the first mimic learning framework for Q functions in DRL. We introduce Linear Model U-trees (LMUTs) to approximate neural network predictions. An LMUT is learned using a novel on-line algorithm that is well-suited for an active play setting, where the mimic learner observes an ongoing interaction between the neural net and the environment. Empirical evaluation shows that an LMUT mimics a Q function substantially better than five baseline methods. The transparent tree structure of an LMUT facilitates understanding the network’s learned strategic knowledge by analyzing feature influence, extracting rules, and highlighting the super-pixels in image inputs. Code related to this paper is available at: https://github.com/Guiliang/uTree_mimic_mountain_car.
Subfields: Artificial Intelligence, Reinforcement Learning, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Lesia Semenova
- Chudi Zhong
- Zhi Chen
- Haiyang Huang
- Kush R. Varshney
- Cynthia Rudin
- Yashesh D. Dhebar
- Subramanya P. Nageshrao
In Reinforcement Learning
- Murray Campbell
- Nicholas Mattei
- Francesca Rossi
- Kalyanmoy Deb
- Erika Puiutta
- Kush R. Varshney
- Djallel Bouneffouf
- Dimitar Filev
In Safety, Robustness & Interpretability
- Lesia Semenova
- Kush R. Varshney
- Zhi Chen
- Chudi Zhong
- Haiyang Huang
- Cynthia Rudin
- Kalyanmoy Deb
- Erika Puiutta
Citations in the corpus
- Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “, posthoc approximations), including tree-based explanations (Coppens et al., 2019; Liu et al., 2018), reward decomposition (Juozapaitis et al.”
- Designing Interpretable Approximations to Deep Reinforcement Learning with Soft Decision Trees (2020, neutral): “In particular, (Liu et al. 2018) explicitly asserts that play performance, the average return achieved by the distilled controller is the most relevant metric.”
- Toward Interpretable-AI Policies Using Evolutionary Nonlinear Decision Trees for Discrete-Action Systems (2020, methodology): “A linear DT-based model is used in [18] to approximate the Q -values of the trained neural network.”
- Interpretable Multi-Objective Reinforcement Learning through Policy Orchestration (2018, background): “Although the policies themselves may not be directly interpretable (though there is recent work in this area (Verma et al. 2018; Liu et al. 2018)), our system does capture the notion of transparency and interpretability as we can see which policy is being applied in real time.”
- Explainable Reinforcement Learning: A Survey (2020, background): “[38], a mimic learning framework based on stochastic gradient descent is introduced.”