Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization

By Michael R. Zhang, T. Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyun Wang, Mohammad Norouzi. 2021

53 citations recorded by the source corpus.

Standard dynamics models for continuous control make use of feedforward computation to predict the conditional distribution of next state and reward given current state and action using a multivariate Gaussian with a diagonal covariance structure. This modeling choice assumes that different dimensions of the next state and reward are conditionally independent given the current state and action and may be driven by the fact that fully observable physics-based simulation environments entail deterministic transition dynamics. In this paper, we challenge this conditional independence assumption and propose a family of expressive autoregressive dynamics models that generate different dimensions of the next state and reward sequentially conditioned on previous dimensions. We demonstrate that autoregressive dynamics models indeed outperform standard feedforward models in log-likelihood on heldout transitions. Furthermore, we compare different model-based and model-free off-policy evaluation (OPE) methods on RL Unplugged, a suite of offline MuJoCo datasets, and find that autoregressive dynamics models consistently outperform all baselines, achieving a new state-of-the-art. Finally, we show that autoregressive dynamics models are useful for offline policy optimization by serving as a way to enrich the replay buffer through data augmentation and improving performance using model-based planning.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Ofir Nachum
3. Pieter Abbeel
4. Chelsea Finn
5. George Tucker
6. Shixiang Shane Gu
7. Aviral Kumar
8. Mohammad Norouzi

In Management Science and Operations Research

1. Sergey Levine
2. Ofir Nachum
3. Aviral Kumar
4. Nan Jiang
5. George Tucker
6. Justin Fu
7. Michael Janner
8. Qiyang Li

In Reinforcement Learning

1. Sergey Levine
2. Ofir Nachum
3. George Tucker
4. Aviral Kumar
5. Shixiang Shane Gu
6. Pieter Abbeel
7. Chelsea Finn
8. Justin Fu

Citations in the corpus

• Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…used, with fully-connected networks parameterizing diagonal-covariance Gaussian distributions being a common choice (Chua et al., 2018), though recent work has highlighted the effectiveness of autoregressive state prediction (Zhang et al., 2021) like that used by the Trajectory Transformer.”
• Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “We follow the setup detailed in Zhang et al. (2021).”
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, background): “[60] Michael R Zhang, Tom Le Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyu Wang, and Mohammad Norouzi.”
• Learning Universal Policies via Text-Guided Video Generation (2023, background): “…environment rewards and dynamics that can serve as “world models” for model-based reinforcement learning and planning have been recently scaled to large-scale architectures developed for vision and language (Hafner et al., 2020; Janner et al., 2021; Zhang et al., 2021; Micheli et al., 2022).”