On Learning Intrinsic Rewards for Policy Gradient Methods

By Zeyu Zheng, Junhyuk Oh, Satinder Singh. 2018

234 citations recorded by the source corpus.

In many sequential decision making tasks, it is challenging to design reward functions that help an RL agent efficiently learn behavior that is considered good by the agent designer. A number of different formulations of the reward-design problem, or close variants thereof, have been proposed in the literature. In this paper we build on the Optimal Rewards Framework of Singh et.al. that defines the optimal intrinsic reward function as one that when used by an RL agent achieves behavior that optimizes the task-specifying or extrinsic reward function. Previous work in this framework has shown how good intrinsic reward functions can be learned for lookahead search based planning agents. Whether it is possible to learn intrinsic reward functions for learning agents remains an open problem. In this paper we derive a novel algorithm for learning intrinsic rewards for policy-gradient based learning agents. We compare the performance of an augmented agent that uses our algorithm to provide additive intrinsic rewards to an A2C-based policy learner (for Atari games) and a PPO-based policy learner (for Mujoco domains) with a baseline agent that uses the same policy learners but with only extrinsic rewards. Our results show improved performance on most but not all of the domains.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Meta-Gradient Reinforcement Learning (2018, methodology): “…corrections [Espeholt et al., 2018, Munos et al., 2016], target networks [Mnih et al., 2015], emphasis on certain states [Sutton et al., 2016], reward clipping [Mnih et al., 2013], or even the nature of the rewards themselves [Randløv and Alstrøm, 1998, Singh et al., 2005, Zheng et al., 2018].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, methodology): “Meta-learning has been repeatedly used in the recent literature to learn functions for intrinsic motivation and improved exploration (Baranes and Oudeyer, 2009; Zheng et al., 2018; Stadie et al., 2018; Xu et al., 2018b; Houthooft et al., 2018; Yang et al., 2019; Zou et al., 2019; Zheng et al., 2019)”
Meta-Gradient Reinforcement Learning with an Objective Discovered Online (2020, unknown): “Algorithm Algorithm properties What is meta-learned? IDBD, SMD [31, 28] → learning rate SGD(2) [1] ← optimiser RL(2), Meta-RL [9, 40] X recurrent network MAML, REPTILE [11, 24] ← initial params Meta-Gradient [44, 47] → γ, λ, reward Meta-Gradient [39, 45, 41] ← auxiliary tasks, hype”
Online Meta-Critic Learning for Off-Policy Actor-Critic Methods (2020, methodology): “For example, while LIRPG (Zheng et al., 2018) is one of the few prior works to attempt online meta-learning, it is ineffective in practice due to only providing a scalar reward increment rather than a loss for direct optimization.”
LIIR: Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning (2019, methodology): “A recent study [16] has investigated learning a parameterized intrinsic reward function in single agent setting.”
What Can Learned Intrinsic Rewards Capture? (2019, methodology): “The chain rule is used to get the meta-gradient (∇ηθt) as in previous work (Zheng et al., 2018).”
Meta-learning curiosity algorithms (2020, background): “Closer to our formulation, Zheng et al. (2018) parametrize an intrinsic reward function which influences policygradient updates in a differentiable manner, allowing them to backpropagate through a single step of the policy-gradient update to optimize the intrinsic reward function for a single task.”
Learning Data Manipulation for Augmentation and Weighting (2019, methodology): “Of particular interest to this work is [52] which learns a parametric intrinsic reward that additively transforms the original task reward (a.”
Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning (2019, background): “[42] studies intrinsic motivation under the optimal reward design framework where the optimal intrinsic reward is learned through gradient descent.”
Self-Tuning Deep Reinforcement Learning (2020, methodology): “In previous work, online metagradients have been used to learn the discount factor or the λ coefficient (Xu et al., 2018), to discover intrinsic rewards (Zheng et al., 2018) and auxiliary tasks (Veeriah et al., 2019).”
Adaptive Auxiliary Task Weighting for Reinforcement Learning (2019, methodology): “Some recent works can be viewed as instantiations of this method, where the meta-parameters are the parameters of the return function [32] or an intrinsic reward function [33].”
Bootstrapped Meta-Learning (2021, background): “…lifetime performance (Lv et al., 2017; Wichrowska et al., 2017), collapsed exploration (Stadie et al., 2018; Chen et al., 2016), biased learner updates (Stadie et al., 2018; Zheng et al., 2018), and poor generalisation performance (Wu et al., 2018; Yin et al., 2020; Triantaﬁllou et al., 2020).”
Learning to Generalize from Sparse and Underspecified Rewards (2019, background): “Another relevant work is LIRPG (Zheng et al., 2018), which learns a parametric intrinsic reward function that can be added to the extrinsic reward to improve the performance of policy gradient methods.”
Discovery of Useful Questions as Auxiliary Tasks (2019, methodology): “Xu et al. (2018) discussed the possibility of computing meta-gradients in a non-myopic manner, but their proposed algorithm, as that of Zheng et al. (2018), introduced a severe approximation and only measured the immediate consequences of an update.”
Discovering Reinforcement Learning Algorithms (2020, methodology): “This approach has been applied to meta-learn various forms of algorithmic components such as the discount factor [38], intrinsic rewards [41], auxiliary tasks [33], returns [36], auxiliary policy updates [42], off-policy corrections [39], and update target [37].”
Provably Efficient Maximum Entropy Exploration (2018, background): “…signal derived from novelty or prediction error [LLTyO12, PAED17, SRM + 18, FCRL17, MJR15, HCC + 16, SRM + 18, WRR + 17], aligning an intrinsic reward to the target objective [Kae93, CBS05, SLBS10, SLB09, ZOS18], or sample based approaches to tracking of value function uncertainty [OBPVR16, OAC18].”
Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisation (2019, background): “Reward learning recently receives increasing interests from the machine learning community [ Ibarz et al. , 2018; Zheng et al. , 2018 ] , but it is largely overlooked in NLG until now.”