What Can Learned Intrinsic Rewards Capture?

By Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, M. Kroiss, Hado van Hasselt, David Silver, Satinder Singh. 2019

88 citations recorded by the source corpus.

The objective of a reinforcement learning agent is to behave so as to maximise the sum of a suitable scalar function of state: the reward. These rewards are typically given and immutable. In this paper, we instead consider the proposition that the reward function itself can be a good locus of learned knowledge. To investigate this, we propose a scalable meta-gradient framework for learning useful intrinsic reward functions across multiple lifetimes of experience. Through several proof-of-concept experiments, we show that it is feasible to learn and capture knowledge about long-term exploration and exploitation into a reward function. Furthermore, we show that unlike policy transfer methods that capture "how" the agent should behave, the learned reward functions can generalise to other kinds of agents and to changes in the dynamics of the environment by capturing "what" the agent should strive to do.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Hado van Hasselt
3. Matteo Hessel
4. David Silver
5. Pieter Abbeel
6. Junhyuk Oh
7. Satinder Singh
8. Zhongwen Xu

In Cognitive Neuroscience

1. Hado van Hasselt
2. Matteo Hessel
3. Jian Peng
4. Satinder Singh
5. Junhyuk Oh
6. David Silver
7. Tanmay Gangwani
8. Pieter Abbeel

In Management Science and Operations Research

1. Hado van Hasselt
2. Matteo Hessel
3. David Silver
4. Zhongwen Xu
5. Satinder Singh
6. Junhyuk Oh
7. Wojciech M. Czarnecki
8. Jian Peng

Citations in the corpus

• Human-Timescale Adaptation in an Open-Ended Task Space (2023, unknown): “, 2021) or history-dependent estimators (Zheng et al., 2020).”
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online (2020, unknown): “The meta-gradient algorithm above can be applied to any differentiable component of the update rule, for example to learn the discount factor γ and bootstrapping factor λ [43], intrinsic rewards [46, 45], and auxiliary tasks [38].”
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning (2021, methodology): “Gradient-based metalearning methods have also been explored (Finn et al., 2017; Nichol & Schulman, 2018; Xu et al., 2018; Zheng et al., 2020).”
• Discovering Reinforcement Learning Algorithms (2020, background): “Discovering Reinforcement Learning Algorithms There have been a few attempts to meta-learn RL algorithms, from earlier work on bandit algorithms [20, 19] to curiosity algorithms [1] and RL objectives [16, 40, 6, 17] (see Table 1 for comparison).”