Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping

By Andrew Y. Ng, Daishi Harada, Stuart J. Russell. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, methodology): “[10], this shaping transformation does not alter the optimal policy (in the discounted setting).”
One-Shot Imitation Learning (2017, background): “…such as behavioral cloning and DAGGER (Ross et al., 2011), which only require demon- strations rather than reward functions to be specified.1 This has the potential to be more scalable, since it is often easier to demonstrate a task than specifying a well-shaped reward function (Ng et al., 1999).”
Scalable agent alignment via reward modeling: a research direction (2018, background): “If the user has some insight into the credit assignment problem, they could use reward shaping (Ng et al., 1999) to teach a reward function that is shaped in the direction of this behavior.”
Learning to Understand Goal Specifications by Modelling Reward (2018, background): “…states within episodes in a manner similar to reward prediction suggests that the reward model within AGILE learns some form of shaped reward (Ng et al., 1999), and could serve not only in the cases where a reward function need to be learned in the absence of true reward, but also in cases…”
Value Iteration Networks (2016, background): “Gray-colored obstacles are randomly positioned in the domain, and apply an elastic force and friction when contacted.”
Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning (2015, background): “he reward signal of an MDP so as to make it more informative. Reward shaping does not alter the optimal policy of an MDP but can make reaching the optimal policy, or at least a local maximum, easier (Ng et al., 1999). Intuitively, instantiating a DQN with a multitask network can be seen as providing”
Embodied Question Answering (2017, background): “As a first step in this challenging space, we judiciously scope out a problem space – environments, question types, and learning paradigm – that allows us to augment sparse RL rewards with imitation learning (showing the agent expert trajectories) and reward shaping [5] (giving intermediate ‘closer/”
Value-Decomposition Networks For Cooperative Multi-Agent Learning (2017, background): “However, even in the single-agent case, reward shaping is difﬁcult and only a small class of shaped reward functions are guaranteed to preserve optimality w.r.t. the true objective (Ng et al., 1999; Devlin et al., 2014; Eck et al., 2016).”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Redistributing 67 the reward is fundamentally different from reward shaping [36, 64], which changes the reward 68 as a function of states but not of actions.”
Dota 2 with Large Scale Deep Reinforcement Learning (2019, background): “Our shaped reward is modeled loosely after potential-based shaping functions [58], though the guarantees therein do not apply here.”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Both D and PBRS have been successfully applied to a wide range of application domains and have the added beneﬁt of convenient theoretical guarantees, meaning that they do not suffer from the same issues as the unprincipled reward shaping approaches described above (see e.g. [49]–[53]).”
Reinforcement Learning in Healthcare: A Survey (2019, background): “reward shaping [234] to facilitate the learning efficiency.”
Reinforcement learning in robotics: A survey (2013, background): “Reward shaping gives the system a notion of closeness to the desired behavior instead of relying on a reward that only encodes success or failure (Ng et al., 1999).”
Disentangling Transfer in Continual Reinforcement Learning (2022, neutral): “[30, 5] apply reward function reshaping.”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “2 The most commonly used CVs for policy gradient [1, 17, 18] are state-dependent functions v̂ : S → R, which leads to the difference estimator G̃t := Gt − ( NtV̂t −EAt|St [NtV̂t] ) = Gt −NtV̂t, where V̂t := v̂(St), (4) and the expectation vanishes as EAt|St [NtV̂t] = V̂t∇EAt|St [1] = 0.”
End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, methodology): “[2], we use reward shaping [13] to deal with this problem.”