Offline Reinforcement Learning with Pseudometric Learning
By Robert Dadashi, Shideh Rezaeifar, Nino Vieillard, L'eonard Hussenot, Olivier Pietquin, Matthieu Geist. 2021
44 citations recorded by the source corpus.
Offline Reinforcement Learning methods seek to learn a policy from logged transitions of an environment, without any interaction. In the presence of function approximation, and under the assumption of limited coverage of the state-action space of the environment, it is necessary to enforce the policy to visit state-action pairs close to the support of logged transitions. In this work, we propose an iterative procedure to learn a pseudometric (closely related to bisimulation metrics) from logged transitions, and use it to define this notion of closeness. We show its convergence and extend it to the function approximation setting. We then use this pseudometric to define a new lookup based bonus in an actor-critic algorithm: PLOFF. This bonus encourages the actor to stay close, in terms of the defined pseudometric, to the support of logged transitions. Finally, we evaluate the method on hand manipulation and locomotion tasks.
Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Josh Tenenbaum
- Yilun Du
- Pieter Abbeel
- Chelsea Finn
- Ruslan Salakhutdinov
- Michael Janner
- Tommi S. Jaakkola
In Control and Systems Engineering
- Sergey Levine
- Josh Tenenbaum
- Pulkit Agrawal
- Michael Janner
- Yilun Du
- Olivier Pietquin
- Qiyang Li
- L'eonard Hussenot
In Reinforcement Learning
- Sergey Levine
- Josh Tenenbaum
- Yilun Du
- Michael Janner
- Qiyang Li
- Olivier Pietquin
- Pieter Abbeel
- Pulkit Agrawal
Citations in the corpus
- Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “Modeling the states and actions jointly already provides a bias toward generating in-distribution actions, which avoids the need for explicit pessimism (Fujimoto et al., 2019; Kumar et al., 2019a; Ghasemipour et al., 2021; Nair et al., 2020; Jin et al., 2021; Yin et al., 2021; Dadashi et al., 2021).”
- RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, background): “From low coverage [11] to full coverage [15] of the state space.”
- Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “Modeling the states and actions jointly already provides a bias toward generating in-distribution actions, which avoids the need for explicit pessimism [16, 31, 17, 42, 27, 66, 12].”
- Is Conditional Generative Modeling all you need for Decision-Making? (2022, background): “Prior works (Kumar et al., 2020; Kostrikov et al., 2022; Wu et al., 2019; Kostrikov et al., 2021; Dadashi et al., 2021; Ajay et al., 2020; Ghosh et al., 2022) have tackled this problem with reinforcement learning (RL) that uses dynamic programming for trajectory stitching.”