Expressing Arbitrary Reward Functions as Potential-Based Advice

By Ani Harutyunyan, Sam Devlin, Peter Vrancx, Ann Nowé. 2015

134 citations recorded by the source corpus.

Effectively incorporating external advice is an important problem in reinforcement learning, especially as it moves into the real world. Potential-based reward shaping is a way to provide the agent with a specific form of additional reward, with the guarantee of policy invariance. In this work we give a novel way to incorporate an arbitrary reward function with the same guarantee, by implicitly translating it into the specific form of dynamic advice potentials, which are maintained as an auxiliary value function learnt at the same time. We show that advice provided in this way captures the input reward function in expectation, and demonstrate its efficacy empirically.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

RUDDER: Return Decomposition for Delayed Rewards (2018, background): “On the other hand every reward function can be expressed as look-ahead advice [43].”
FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, background): “o encode ‘rules’ of the environment of the RL agent. However, potential functions will typically need to be pre-specified. This has restricted the use of PBRS to tabular/ low-dimensional state spaces [15, 38]. The cycling problem (repeatedly visiting certain states) mention in Section 4.1 can be res”
Learning from Demonstration for Shaping through Inverse Reinforcement Learning (2016, background): “An extensively explored way to provide denser information is through shaping the existing reward function by adding an extra signal, thus providing the agent with a composite reward [14, 6, 4, 8, 25].”
Autonomous Navigation of UAVs in Large-Scale Complex Environments: A Deep Reinforcement Learning Approach (2019, background): “Nonetheless, since it only makes sense under the condition that the shaped reward is the difference of a state-dependent potential function, it is hard to implement in practice and tends to block the way of incorporating domain knowledge [40].”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, methodology): “[76] ( This leads to the approach of using the negation of R + as the immediate reward to train an extra state-action Value function Φ and the policy simultaneously.”
Sidekick Policy Learning for Active Visual Exploration (2018, background): “Potential-based reward shaping [23] incorporates expert knowledge grounded in potential functions to ensure policy invariance.”
Policy Transfer using Reward Shaping (2015, background): “[8] combine these two extensions into dynamic shaping over state-action pairs:”
What Can Learned Intrinsic Rewards Capture? (2019, background): “Both the beneﬁts (Randl¨ov & Al-str¨om, 1998; Ng et al., 1999; Harutyunyan et al., 2015) and the difﬁculty (Clark & Amodei, 2016) of task-speciﬁc reward shaping have been studied.”