Inverse Reward Design

By Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J. Russell, Anca Dragan. 2017

495 citations recorded by the source corpus.

Autonomous agents optimize the reward function we give them. What they don't know is how hard it is for us to design a reward function that actually captures what we want. When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios. Inevitably, agents encounter new scenarios (e.g., new types of terrain) where optimizing that same reward may lead to undesired behavior. Our insight is that reward functions are merely observations about what the designer actually wants, and that they should be interpreted in the context in which they were designed. We introduce inverse reward design (IRD) as the problem of inferring the true objective based on the designed reward and the training MDP. We introduce approximate methods for solving IRD problems, and use their solution to plan risk-averse behavior in test MDPs. Empirical results suggest that this approach can help alleviate negative side effects of misspecified reward functions and mitigate reward hacking.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Industrial and Manufacturing Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Industrial and Manufacturing Engineering

In Reinforcement Learning

Citations in the corpus

Alignment of Language Agents (2021, background): “Another class of approaches (Hadfield-Menell et al., 2016, 2017b) contains an agent which is uncertain about its objective, and aims for the agent to correctly calibrate its beliefs about it, and in doing so avoid gaming it.”
Scalable agent alignment via reward modeling: a research direction (2018, background): “…Asri et al., 2016), actions (Grifﬁth et al., 2013), value (Knox & Stone, 2009), advantage (MacGlashan et al., 2017), or preferences over trajectories (Fürnkranz et al., 2012; Akrour et al., 2012, 2014; Wirth et al., 2017); and providing an explicit objective function (Hadﬁeld-Menell et al., 2017b).”
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “Hadﬁeld-Menell et al. (2017) show and mitigate the reward hacking that arises due to an incorrect sensor reading at test time in a 10x10 navigation grid world.”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “Note that a feature-based reward function is a standard assumption in the literature of inverse RL (Ng et al., 2000; Ziebart et al., 2008; Hadfield-Menell et al., 2017).”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “While we can take the human out of the loop by designing a reward function, it is challenging to design a reward function that elicits the desired behaviors from the agent (Hadﬁeld-Menell et al., 2017).”
Playing hard exploration games by watching YouTube (2018, background): “Despite the challenges of designing reward functions [18, 36] or learning them using inverse reinforcement learning [1, 49], we also achieve human-level performance even in the absence of an environment reward signal.”
Occam's razor is insufficient to infer the preferences of irrational agents (2017, background): “The environment M in which the human operates is an MDP/R, a Markov Decision Process without reward function (a world-model [Hadﬁeld-Menell et al., 2017]).”
AI Safety Gridworlds (2017, methodology): “eciﬁed in a small fraction of all states, adopting a robust choice strategy signiﬁcantly reduces regret. This method works by combining a “good-enough” reward level with randomization (Taylor, 2016). Hadﬁeld-Menell et al. (2017) instead rely on inference of the intended behavior from the speciﬁed re”
Hidden Incentives for Auto-Induced Distributional Shift (2020, background): “a POMDP with no reward function, also known as a world model (Armstrong & O’Rourke, 2017; Hadfield-Menell et al., 2017); the difference is that the learner observes the input (opre) before acting and only observes the target (opost) after acting.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “Other strategies can include forms of risk-aversion [Hadfield-Menell et al., 2017] or handling uncertainty with a safe “shield” policy [Jansen et al.”
Bayesian Robust Optimization for Imitation Learning (2020, methodology): “Alternatively the distribution could be the posterior distribution P(R | D) learned via Bayesian inverse reinforcement learning [43] given demonstrations D or the posterior distribution P(R | R′) learned via inverse reward design given a human-specified proxy reward function R′ [23].”
Reward-rational (implicit) choice: A unifying formalism for reward learning (2020, background): “In “oﬀ” feedback [Hadﬁeld-Menell et al., 2017a], the robot executes a trajectory, and at any point, the human may switch the robot oﬀ.”
Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning (2019, background): “Inverse reward design [10] studies the opposite problem of inferring true reward from designed reward.”
Reward (Mis)design for Autonomous Driving (2021, background): “Methods of learning reward functions include inverse reinforcement learning from demonstrations [35, 58], learning reward functions from pairwise preferences over trajectories [55, 9, 6], and inverse reward design from trial-and-error reward design in multiple instances of a task domain [18].”