Hyperbolic Discounting and Learning over Multiple Horizons

By William Fedus, Carles Gelada, Yoshua Bengio, Marc G. Bellemare, Hugo Larochelle. 2019

123 citations recorded by the source corpus.

Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences. In this work we revisit the fundamentals of discounting in RL and bridge this disconnect by implementing an RL agent that acts via hyperbolic discounting. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over a strong value-based RL agent, Rainbow.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Mathematical Physics, Game & Decision Theory, Economics & Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Pieter Abbeel
3. Yoshua Bengio
4. Ruslan Salakhutdinov
5. Quoc V. Le
6. Yuxi Li
7. David Silver
8. Chelsea Finn

In Computational Theory and Mathematics

1. Marc G. Bellemare
2. Sergey Levine
3. Ofir Nachum
4. Carles Gelada
5. David Silver
6. Hado van Hasselt
7. Junhyuk Oh
8. Satinder Singh

In Mathematical Physics

1. Mark Rowland
2. Yoshua Bengio
3. Marc G. Bellemare

In Game & Decision Theory

1. Junhyuk Oh
2. Satinder Singh
3. Mark Rowland
4. David Silver
5. Yoshua Bengio
6. Marc G. Bellemare
7. Sylvain Gelly
8. Will Dabney

In Economics & Operations Research

1. Carlos Riquelme
2. David Silver
3. Yoshua Bengio
4. Hado van Hasselt

Citations in the corpus

• DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “On the other hand, many recently-proposed approaches to representation learning via deep learning have strong empirical results on complex domains, but lack formal guarantees (Jaderberg et al., 2016; van den Oord et al., 2018; Fedus et al., 2019).”
• Scalable Transfer Learning with Expert Models (2020, background): “Among others, this has been successfully applied to vision [72], language [34], and reinforcement learning [18] tasks.”
• Self-Tuning Deep Reinforcement Learning (2020, methodology): “Inspired by Fedus et al. (2019), STACX augments STAC with parametric auxiliary loss functions (each with it’s own hyper-parameters).”
• Revisiting Fundamentals of Experience Replay (2020, background): “, 2019), learning algorithms using auxiliary tasks (Jaderberg et al., 2016; Fedus et al., 2019), distributional variants of RL (Bellemare et al.”