Model-Based Offline Planning

By Arthur Argenson, Gabriel Dulac-Arnold. 2020

172 citations recorded by the source corpus.

Offline learning is a key part of making reinforcement learning (RL) useable in real systems. Offline RL looks at scenarios where there is data from a system's operation, but no direct access to the system when learning a policy. Recent work on training RL policies from offline data has shown results both with model-free policies learned directly from the data, or with planning on top of learnt models of the data. Model-free policies tend to be more performant, but are more opaque, harder to command externally, and less easy to integrate into larger systems. We propose an offline learner that generates a model that can be used to control the system directly through planning. This allows us to have easily controllable policies directly from data, without ever interacting with the system. We show the performance of our algorithm, Model-Based Offline Planning (MBOP) on a series of robotics-inspired tasks, and demonstrate its ability leverage planning to respect environmental constraints. We are able to find near-optimal polices for certain simulated systems from as little as 50 seconds of real-time system interaction, and create zero-shot goal-conditioned policies on a series of environments.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…well, such as ensembles in the online setting (Kurutach et al., 2018; Buckman et al., 2018; Malik et al., 2019) or conservatism mechanisms in the ofﬂine setting (Yu et al., 2020; Kidambi et al., 2020; Argenson & Dulac-Arnold, 2021), our method does not require explicit handling of these components.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “It is relatively common in prior work on model-based offline RL to select various hyperparameters using online policy rollouts [67, 26, 3, 33].”
Representation Balancing Offline Model-based Reinforcement Learning (2021, methodology): “Matsushima et al. (2020), MOOSE (Swazinna et al., 2020) and MBOP (Argenson & Dulac-Arnold, 2020) guide their policy optimization using the behavior policy, similar to the behavior regularization approaches.”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, methodology): “Recent offline RL methods use many algorithmic techniques, including value functions [15, 27], dynamics estimation alongisde value functions [24, 37, 42], dynamics estimation alone [3, 7, 39], and uncertainty quantification [1, 41, 42].”
Model-Based Offline Planning with Trajectory Pruning (2021, unknown): “, 2020], as well as the recent model-based offline planning algorithm MBOP [Argenson and Dulac-Arnold, 2021].”
Offline Reinforcement Learning from Images with Latent Space Models (2020, background): “…al., 2019), approximate dynamic programming algorithms (Fujimoto et al., 2018; Kumar et al., 2019, 2020; Liu et al., 2020; Agarwal et al., 2019), and model-based RL algorithms (Kidambi et al., 2020; Yu et al., 2020b; Argenson and Dulac-Arnold, 2020; Matsushima et al., 2020; Swazinna et al., 2020).”
Offline Reinforcement Learning with Pseudometric Learning (2021, background): “As our bonus is based on learned structural properties of the MDP we can also draw a connection to model-based off-policy RL (Kidambi et al., 2020; Yu et al., 2020; Argenson & Dulac-Arnold, 2021).”
Temporal Difference Learning for Model Predictive Control (2022, methodology): “…a learned policy, but it can be prohibitively expensive to plan over long horizons (Janner et al., 2019; Lowrey et al., 2019; Hafner et al., 2019; Argenson & Dulac-Arnold, 2021); and (ii) using a learned model to improve sample-efﬁciency of model-free methods by e.g. learning from generated…”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “We compare against four other methods: (1) conservative Q-learning (CQL; [33]), (2) model-based offline policy optimization (MOPO; [67]), model-based offline planning (MBOP; [3]), and behavior cloning (BC).”
Planning with Diffusion for Flexible Behavior Synthesis (2022, unknown): “…compare to a variety of prior algorithms spanning other approaches to data-driven control, including the model-free reinforcement learning algorithms CQL (Kumar et al., 2020) and IQL (Kostrikov et al., 2022) (Yu et al., 2020), MOReL (Kidambi et al., 2020), and MBOP (Argenson & Dulac-Arnold, 2021).”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, methodology): “Works such as MOPO (Yu et al. 2020) and MoREL (Kidambi et al. 2020) leverage the learnt model to learn a model-free policy, and approaches such as MBOP (Argenson and Dulac-Arnold 2020) leverage the model directly using an MPC-based planner.”