Improving Robot Controller Transparency Through Autonomous Policy Explanation

By Bradley Hayes, Julie Arnold Shah. 2017

334 citations recorded by the source corpus.

Shared expectations and mutual understanding are critical facets of teamwork. Achieving these in human-robot collaborative contexts can be especially challenging, as humans and robots are unlikely to share a common language to convey intentions, plans, or justifications. Even in cases where human co-workers can inspect a robot's control code, and particularly when statistical methods are used to encode control policies, there is no guarantee that meaningful insights into a robot's behavior can be derived or that a human will be able to efficiently isolate the behaviors relevant to the interaction. We present a series of algorithms and an accompanying system that enables robots to autonomously synthesize policy descriptions and respond to both general and targeted queries by human collaborators. We demonstrate applicability to a variety of robot controller types including those that utilize conditional logic, tabular reinforcement learning, and deep reinforcement learning, synthesizing informative policy descriptions for collaborators and facilitating fault diagnosis by non-experts.

Subfields: Artificial Intelligence, Information Systems, Safety, Robustness & Interpretability, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Safety, Robustness & Interpretability

In Robotics & Control

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “…(i.e., posthoc approximations), including tree-based explanations (Coppens et al., 2019; Liu et al., 2018), reward decomposition (Juozapaitis et al., 2019), autonomous explanation (Fukuchi et al., 2017; Hayes and Shah, 2017), and attention-based methods (Zambaldi et al., 2018; Mott et al., 2019).”
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, background): “…Amodei, et al., 2016; Belotti and Edwards, 2001; Bostrom andYudkowsky, 2014; Dwork, et al., 2012; Fallon and Blaha, 2018; Hajian, et al., 2015; Hayes and Shah, 2017; Joseph, et al., 2016a,b; Kroll, et al., 2016; Lum and Isaac, 2016; Otte, 2013; Sweeney, 2013; Tate, et al., 2016; Varsheny and…”
Explanation in Artificial Intelligence: Insights from the Social Sciences (2017, background): “Applications of explanation are considered in many sub-fields of artificial intelligence, such as justifying autonomous agent behaviour [129, 65], debugging of machine learning models [89], explaining medical decision-making [45], and explaining predictions of classifiers [156].”
Visualizing and Understanding Atari Agents (2017, background): “More recently there has been work on explaining RL agents that does not require an MDP model [7].”
Counterfactual States for Atari Agents via Generative Deep Learning (2019, background): “…work for explaining RL has looked at explaining policies through t-SNE embeddings [ Mnih and Hassabis, 2015; Khan et al. , 2009 ] , state abstractions [ Zahavy et al. , 2016 ] , human-interpretable predicates [ Hayes and Shah, 2017 ] and a high-level programming language [ Verma et al. , 2018 ] .”
Learn to Interpret Atari Agents (2018, background): “Traditionally, interpreting RL systems may involve language generation via ﬁrst-order logic [28]–[31].”
Evaluating XAI: A comparison of rule-based and example-based explanations (2021, methodology): “…way the advice is contrasted with a different advice, for example using rules or examples Within the context of a DSS advising an insulin dose for DMT1 self-management, a contrastive rule-based explanation could be: “Currently the temperature is below 10 degrees and a lower insulin dose is advised.”
Explainable Reinforcement Learning: A Survey (2020, neutral): “[66]) • Autonomous Policy Explanation (Hayes and Shah [21])”
Will You Accept an Imperfect AI?: Exploring Designs for Adjusting End-user Expectations of AI Systems (2019, background): “These works automatically select informative examples prior to actual system use [2, 23, 40] to give the user an intuition about the system behavior or attempt to summarize internal system decision rules [21].”
Explainable Sentiment Analysis with Applications in Medicine (2018, background): “The generated explanation, given in terms of state-actions and rewards, are translated into a more descriptive state classes and outcomes by following the approaches of [61] and [62].”
Explainable Agents and Robots: Results from a Systematic Literature Review (2019, background): “, [44]) and teaming for military missions [104] are the particularly preferred cases of the surveyed articles.”
Bayes-Probe: Distribution-Guided Sampling for Prediction Level Sets (2020, background): “One such policy summarization approach formulates a set cover problem over histories of system states to produce an overview of an agent’s behaviour (Hayes & Shah, 2017).”
Explanations in Autonomous Driving: A Survey (2021, background): “The explainable planning frameworks described above and the related work by [90], [137], [139], [140] can serve as”
Reasons, Values, Stakeholders: A Philosophical Framework for Explainable Artificial Intelligence (2021, background): “1 More concretely, the development of explainable AI systems is motivated by several social and ethical reasons (see, for example, [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]): increasing societal acceptance of prediction-based decisions, establishing trust in the results of these decisions, making…”
Transparency you can trust: Transparency requirements for artificial intelligence between legal norms and contextual concerns (2019, background): “While achieving a full understanding of information technologies is typically diﬃcult due to their complexity (Hayes and Shah, 2017), transparent explanations of, for example, reasons for robot behavior, can contribute to an increased perception of their trustworthiness (Korpan et al., 2017;…”