Policy Gradients Incorporating the Future
By David Venuto, Elaine Lau, Doina Precup, Ofir Nachum. 2021
9 citations recorded by the source corpus.
Reasoning about the future -- understanding how decisions in the present time affect outcomes in the future -- is one of the central challenges for reinforcement learning (RL), especially in highly-stochastic or partially observable environments. While predicting the future directly is hard, in this work we introduce a method that allows an agent to"look into the future"without explicitly predicting it. Namely, we propose to allow an agent, during its training on past experience, to observe what \emph{actually} happened in the future at that time, while enforcing an information bottleneck to avoid the agent overly relying on this privileged information. This gives our agent the opportunity to utilize rich and useful information about the future trajectory dynamics in addition to the present. Our method, Policy Gradients Incorporating the Future (PGIF), is easy to implement and versatile, being applicable to virtually any policy gradient algorithm. We apply our proposed method to a number of off-the-shelf RL algorithms and show that PGIF is able to achieve higher reward faster in a variety of online and offline RL domains, as well as sparse-reward and partially observable environments.
Subfields: Artificial Intelligence, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Doina Precup
- Marc G. Bellemare
- R´emi Munos
- Yoshua Bengio
- Nicolas Manfred Otto Heess
- Joelle Pineau
In Management Science and Operations Research
- Doina Precup
- Nicolas Manfred Otto Heess
- R´emi Munos
- Marc G. Bellemare
- Will Dabney
- Sergey Levine
- Arthur Guez
- Theophane Weber
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “As a result, replay has become a very successful approach for tackling continual RL (Isele and Cosgun, 2018; Riemer et al., 2019; Rolnick et al., 2019; Oh et al., 2021; Henning et al., 2021; Liu et al., 2021b; Queeney et al., 2021; Chandak et al., 2021; Lampinen et al., 2021; Venuto et al., 2021; Li”
- Counterfactual Credit Assignment in Model-Free Reinforcement Learning (2020, methodology): “, 2016), Generative CCA is very similar to the approach proposed by (Venuto et al., 2021).”