Reward (Mis)design for Autonomous Driving
By W. B. Knox, Alessandro Gabriele Allievi, Holger Banzhaf, Felix Schmitt, Peter Stone. 2021
169 citations recorded by the source corpus.
This article considers the problem of diagnosing certain common errors in reward design. Its insights are also applicable to the design of cost functions and performance metrics more generally. To diagnose common errors, we develop 8 simple sanity checks for identifying flaws in reward functions. We survey research that is published in top-tier venues and focuses on reinforcement learning (RL) for autonomous driving (AD). Specifically, we closely examine the reported reward function in each publication and present these reward functions in a complete and standardized format in the appendix. Wherever we have sufficient information, we apply the 8 sanity checks to each surveyed reward function, revealing near-universal flaws in reward design for AD that might also exist pervasively across reward design for other tasks. Lastly, we explore promising directions that may aid the design of reward functions for AD in subsequent research, following a process of inquiry that can be adapted to other domains.
Subfields: Artificial Intelligence, Cognitive Neuroscience, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Kush Bhatia
- Dorsa Sadigh
- Alexander Pan
- Peng Wu
- Bernhard Jaeger
- Li Chen
- Kalesha Bullard
- Kashyap Chitta
In Cognitive Neuroscience
In Reinforcement Learning
- Kush Bhatia
- Dorsa Sadigh
- Alexander Pan
- Peng Wu
- Bernhard Jaeger
- Li Chen
- Kalesha Bullard
- Kashyap Chitta
Citations in the corpus
- The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “Reward hacking , or the gaming of misspecified reward functions by RL agents, has appeared in a variety of contexts, such as game playing (Ibarz et al., 2018), text summarization (Paulus et al., 2018), and autonomous driving (Knox et al., 2021).”
- End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “[48], [80], [81].”
- Reward Design with Language Models (2023, background): “Reward hacking has appeared in various domains such as autonomous driving (Knox et al., 2021) and game-playing (Ibarz et al., 2018).”