Reinforcement Learning with a Corrupted Reward Channel

By Tom Everitt, Victoria Krakovna, Laurent Orseau, Shane Legg. 2017

130 citations recorded by the source corpus.

No real-world reward function is perfect. Sensory errors and software bugs may result in RL agents observing higher (or lower) rewards than they should. For example, a reinforcement learning agent may prefer states where a sensory error gives it the maximum reward, but where the true reward is actually small. We formalise this problem as a generalised Markov Decision Problem called Corrupt Reward MDP. Traditional RL methods fare poorly in CRMDPs, even under strong simplifying assumptions and when trying to compensate for the possibly corrupt rewards. Two ways around the problem are investigated. First, by giving the agent richer data, such as in inverse reinforcement learning and semi-supervised reinforcement learning, reward corruption stemming from systematic sensory errors may sometimes be completely managed. Second, by using randomisation to blunt the agent's optimisation, reward corruption can be partially managed under some assumptions.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “This requires the reward model to be accurate off-policy , i.e. on states the agent has never visited (Everitt et al., 2017).”
Scaling Laws for Reward Model Overoptimization (2022, neutral): “Many approaches have been proposed for reducing overoptimization in general [Taylor, 2016, Everitt et al., 2017], as well as in RMs [Gleave and Irving, 2022], including within the ﬁeld of adversarial robustness [Chakraborty et al., 2018].”
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “Everitt et al. (2017) theoretically study and mitigate reward hacking caused by a faulty sensor.”
Let's Verify Step by Step (2023, background): “In the worst case, the use of outcomes as an imperfect proxy could lead to models that become misaligned after learning to exploit the reward signal (Uesato et al., 2022; Cotra, 2022; Everitt et al., 2017).”
Embedded Agency (2019, background): “The lesson we want to draw from the above is from “Reinforcement Learning with a Corrupted Reward Channel” (Everitt et al. 2017): the way that one sets up 8.”
Reward Estimation for Variance Reduction in Deep Reinforcement Learning (2018, methodology): “We further define a corrupted reward to be such that the provided reward does not match the true reward due to some noise process, similarly to the Corrupted Reward MDP (CRMDP) setting of [2].”
Solving math word problems with process- and outcome-based feedback (2022, unknown): “A common concern within the AI safety literature is from RL agents which tamper (Everitt et al., 2017), i.e., corrupt their feedback mechanisms in order to receive positive feedback.”
Human-Centered Artificial Intelligence and Machine Learning (2019, unknown): “Commonsense failure goals occur when an intelligent agent does not achieve the desired result because part of the goal, or the way the goal should have been achieved, is left unstated (this is also referred to as a corrupted goal or corrupted reward (Everitt et al., 2017)).”
Defining and Characterizing Reward Hacking (2022, unknown): “Reward tampering (Everitt et al., 2017; Kumar et al., 2020; Uesato et al., 2020; Everitt et al., 2021) can be viewed as a special case of reward hacking, and refers to an agent corrupting the process generating reward signals, e.g. by tampering with sensors, memory registers storing the reward…”
The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches (2018, unknown): “Q-learning is defined as a model-free reinforcement learning approach which is used to find an optimal action-selection policy for any given (finite) Markov Decision Process (MDP).”