Explainable Reinforcement Learning Through a Causal Lens

By Prashan Madumal, Tim Miller, Liz Sonenberg, Frank Vetere. 2019

422 citations recorded by the source corpus.

Prominent theories in cognitive science propose that humans understand and represent the knowledge of the world through causal relationships. In making sense of the world, we build causal models in our mind to encode cause-effect relations of events and use these to explain why new events happen by referring to counterfactuals — things that did not happen. In this paper, we use causal models to derive causal explanations of the behaviour of model-free reinforcement learning agents. We present an approach that learns a structural causal model during reinforcement learning and encodes causal relationships between variables of interest. This model is then used to generate explanations of behaviour based on counterfactual analysis of the causal model. We computationally evaluate the model in 6 domains and measure performance and task prediction accuracy. We report on a study with 120 participants who observe agents playing a real-time strategy game (Starcraft II) and then receive explanations of the agents' behaviour. We investigate: 1) participants' understanding gained by explanations through task prediction; 2) explanation satisfaction and 3) trust. Our results show that causal model explanations perform better on these measures compared to two other baseline explanation models.

Subfields: Artificial Intelligence, General Decision Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Decision Sciences

Citations in the corpus

Causal Interpretability for Machine Learning - Problems, Methods and Evaluation (2020, methodology): “”For a face generator, is there an internal encoding of the eyes, independent of the remaining facial features?”, by manipulating the internal variables using counterfactual inference. Madumal et al. [68] leverage causal inference to explain the behavior of reinforcement learning agents by learning ”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “Alternatively, one can expect to generate explanations in more interactive settings, such as for the actions and policies of a reinforcement learning agent [115, 116, 172] or for recommender”
FOCUS: Flexible Optimizable Counterfactual Explanations for Tree Ensembles (2019, background): “Counterfactual examples have been used in a variety of ML areas, such as reinforcement learning [23], deep learning [1], and explainable AI (XAI) [8, 13, 17, 18, 22, 24, 26, 28, 31, 35, 36].”
Explainable Reinforcement Learning: A Survey (2020, positive): “al. [51]) • Linear Model U-Trees(Liu et al. [38]) • Interestingness Elements (Sequeira and Gervasio [53]) • Autonomous Self-Explanation (Fukuchi et al. [17]) • Structural Causal Model (Madumal et al. [41]) • Complementary RL (Lee [33]) • Expected Consequences (van der Waa et al. [64]) • Soft Decisio”
Explainable AI Methods - A Brief Overview (2020, methodology): “Description: In the work of Madumal et al. [53] a structural causal model [29] is learned, which can be considered an extension of Bayesian Models [44], [71] of the RL environment with the use of counterfactuals.”
Acquisition of chess knowledge in AlphaZero (2021, methodology): “Structural causal models have been used in learning the action influence models [55], to answer counterfactual questions about the actions taken.”
Aligning Faithful Interpretations with their Social Attribution (2020, background): “work has connected an accurate (faithful) attribution of causality with increased explainability (Feder et al., 2020; Madumal et al., 2020).”
Sequential Explanations with Mental Model-Based Policies (2020, background): “Other studies used (or supported the use of) similar metrics as proxies for inter-pretability (Doshi-Velez & Kim, 2017; Lipton, 2016; Gilpin et al., 2018; Madumal et al., 2019; Poursabzi-Sangdeh et al., 2018; Lahav et al., 2018).”
Explainability in Deep Reinforcement Learning (2020, background): “Thus, the low-level agent will have a succession of sub-goals (e.g. multiple actions that the robotic arm must perform such as moving or opening its pinch) that will ultimately lead to the achievement of the high-level goal (i.e. grasping the red ball).”