Explainable Agency for Intelligent Autonomous Systems
By Pat Langley, B. Meadows, M. Sridharan, Dongkyu Choi. 2017
198 citations recorded by the source corpus.
As intelligent agents become more autonomous, sophisti- cated, and prevalent, it becomes increasingly important that humans interact with them effectively. Machine learning is now used regularly to acquire expertise, but common techniques produce opaque content whose behavior is difficult to interpret. Before they will be trusted by humans, autonomous agents must be able to explain their decisions and the reasoning that produced their choices. We will refer to this general ability as explainable agency. This capacity for explaining decisions is not an academic exercise. When a self-driving vehicle takes an unfamiliar turn, its passenger may desire to know its reasons. When a synthetic ally in a computer game blocks a player’s path, he may want to understand its purpose. When an autonomous military robot has abandoned a high-priority goal to pursue another one, its commander may request justification. As robots, vehicles, and synthetic characters become more selfreliant, people will require that they explain their behaviors on demand. The more impressive these agents’ abilities, the more essential that we be able to understand them.
Subfields: Artificial Intelligence, LLM Agents & Tool Use, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Longo Luca
- Giulia Vilone
- Derek Long
- Maria Fox
- Daniele Magazzeni
- Christian Walter Peter Omlin
- Waddah Saeed
- Chunyan Miao
In LLM Agents & Tool Use
In Safety, Robustness & Interpretability
- Longo Luca
- Giulia Vilone
- Christian Walter Peter Omlin
- Waddah Saeed
- Qiang Yang
- Mohammed Berrada
- Amina Adadi
- Gesina Schwalbe
Citations in the corpus
- Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “• Interactivity: Some contributions [50,59] include the ability of a model to be interactive with the user as one of the goals targeted by an explainable ML model.”
- Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, background): “Recent works have investigated the possibilities of explainability of advanced autonomous agents like socially interactive robots [175]–[178].”
- Explainable Planning (2017, methodology): “Pat Langley et al. more recently used Explainable Agency to refer to the ability of autonomous agents to explain their decisions, and in [Langley et al., 2017] they discuss some functions that agents should exhibit.”
- Building Ethics into Artificial Intelligence (2018, background): “Argumentation-based explainable AI [Fan and Toni, 2015; Langley et al. , 2017] can be a good starting point for this purpose as it is well suited to the consequentialist ethics which is a commonly adopted approach for implementing AI ethics.”
- Explainable Artificial Intelligence: a Systematic Review (2020, background): “The explanations of task planning systems, according to [12, 128], must contain information on (I) why a planner choose an action, (II) why a planner did not choose another action, (III) why the decisions of a planner are the best among a set of possible alternatives, (IV) why certain actions cannot”
- How People Explain Action (and Autonomous Intelligent Systems Should Too) (2017, background): “Explainable AIS should therefore have the ability to clarify their actions by offering the reasons for those actions—the goals, beliefs, duties, and so forth, that motivate and justify the actions (Langley et al. 2017; Broekens et al. 2010).”
- Evaluating Explanations by Cognitive Value (2018, unknown): “There are several other impressive works across various fields catered towards helping the AI engineers [11, 13, 15, 16, 29–32, 40, 38].”
- Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities (2021, background): “Explainable agency refers to a general capability in which autonomous agents must provide explanations for their decisions and the reasons leading to these decisions [116].”
- Explainable Agents and Robots: Results from a Systematic Literature Review (2019, background): “, robots) explaining their actions and the reasons leading to their decisions [60].”