A Game Theoretic Framework for Model Based Reinforcement Learning

By Aravind Rajeswaran, Igor Mordatch, Vikash Kumar. 2020

141 citations recorded by the source corpus.

Model-based reinforcement learning (MBRL) has recently gained immense interest due to its potential for sample efficiency and ability to incorporate off-policy data. However, designing stable and efficient MBRL algorithms using rich function approximators have remained challenging. To help expose the practical challenges in MBRL and simplify algorithm design from the lens of abstraction, we develop a new framework that casts MBRL as a game between: (1) a policy player, which attempts to maximize rewards under the learned model; (2) a model player, which attempts to fit the real-world data collected by the policy player. For algorithm development, we construct a Stackelberg game between the two players, and show that it can be solved with approximate bi-level optimization. This gives rise to two natural families of algorithms for MBRL based on which player is chosen as the leader in the Stackelberg game. Together, they encapsulate, unify, and generalize many previous MBRL algorithms. Furthermore, our framework is consistent with and provides a clear basis for heuristics known to be important in practice from prior works. Finally, through experiments we validate that our proposed algorithms are highly sample efficient, match the asymptotic performance of model-free policy gradient, and scale gracefully to high-dimensional tasks like dexterous hand manipulation.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Management Science and Operations Research

Citations in the corpus

MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “The practical instantiation implements the USAD using an ensemble of randomly initialized dynamics models and plans in the MDP with familiar policy gradient methods like NPG [32, 33, 28].”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “…as an ensemble (Chua et al., 2018; Kurutach et al., 2018; Janner et al., 2019), meta-learning (Clavera et al., 2018), energy-based regularizer (Boney et al., 2019), game-theoretic framework (Rajeswaran et al., 2020), and explicit penalty for unknown states (Kidambi et al., 2020; Yu et al., 2020).”
Model-augmented Prioritized Experience Replay (2022, methodology): “Thus far, various MbRL methods (Kurutach et al., 2018; Luo et al., 2018; Clavera et al., 2018; Janner et al., 2019; Rajeswaran et al., 2020; Clavera et al., 2020; Silver et al., 2017; Schrittwieser et al., 2020; Hafner et al., 2020a; Kaiser et al., 2019) have been proposed, but the common strategy…”
Model-based Adversarial Meta-Reinforcement Learning (2020, methodology): “One popular branch in MBRL is Dyna-style algorithms (Sutton, 1990), which iterates between collecting samples for model update and improving the policy with virtual data generated by the learned model (Luo et al., 2018; Janner et al., 2019; Wang & Ba, 2019; Chua et al., 2018; Buckman et al., 2018; K”
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (2021, background): “, 2020), and be analyzed through new theoretical frameworks (Rajeswaran et al., 2020; Dong et al., 2020).”
Offline Reinforcement Learning from Images with Latent Space Models (2020, background): “Model-based RL algorithms have demonstrated impressive sample efficiency in interactive RL (Janner et al., 2019; Rajeswaran et al., 2020; Hafner et al., 2020).”