Causal Analysis of Agent Behavior for AI Safety
By Gr'egoire D'eletang, Jordi Grau-Moya, Miljan Martic, Tim Genewein, Tom McGrath, Vladimir Mikulik, Markus Kunesch, Shane Legg, Pedro A. Ortega. 2021
8 citations recorded by the source corpus.
As machine learning systems become more powerful they also become increasingly unpredictable and opaque. Yet, finding human-understandable explanations of how they work is essential for their safe deployment. This technical report illustrates a methodology for investigating the causal mechanisms that drive the behaviour of artificial agents. Six use cases are covered, each addressing a typical question an analyst might ask about an agent. In particular, we show that each question cannot be addressed by pure observation alone, but instead requires conducting experiments with systematically chosen manipulations so as to generate the correct causal evidence.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Ruslan Salakhutdinov
- Chelsea Finn
- Quoc V. Le
- J. Urgen Schmidhuber
- Trevor Darrell
In Safety, Robustness & Interpretability
- Shane Legg
- Marcus Hutter
- Pedro A. Ortega
- Markus Kunesch
- Jordi Grau-Moya
- Scott E. Reed
- Nando de Freitas
- Tim Genewein
Citations in the corpus
- Shaking the foundations: delusions in sequence models for interaction and control (2021, background): “2The self-delusions discussed here are unrelated to the problem arising from observational confounding [2, 8, 25].”