Task-Agnostic Dynamics Priors for Deep Reinforcement Learning

By Yilun Du, Karthik Narasimhan. 2019

34 citations recorded by the source corpus.

While model-based deep reinforcement learning (RL) holds great promise for sample efficiency and generalization, learning an accurate dynamics model is often challenging and requires substantial interaction with the environment. A wide variety of domains have dynamics that share common foundations like the laws of classical mechanics, which are rarely exploited by existing algorithms. In fact, humans continuously acquire and use such dynamics priors to easily adapt to operating in new environments. In this work, we propose an approach to learn task-agnostic dynamics priors from videos and incorporate them into an RL agent. Our method involves pre-training a frame predictor on task-agnostic physics videos to initialize dynamics models (and fine-tune them) for unseen target environments. Our frame prediction architecture, SpatialNet, is designed specifically to capture localized physical phenomena and interactions. Our approach allows for both faster policy learning and convergence to better policies, outperforming competitive approaches on several different environments. We also demonstrate that incorporating this prior allows for more effective transfer between environments.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “Conversely, model rollouts may be used to provide extra training examples for a Q-function (Sutton, 1990), to improve the target value estimates of existing data points (Feinberg et al., 2018), or to provide additional context to a policy (Du & Narasimhan, 2019).”
ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, background): “Recent papers use neural networks over visual inputs to predict future pixels (Finn et al., 2016; Lerer et al., 2016; Mirza et al., 2016; Du and Narasimhan, 2019) or make qualitative predictions (Groth et al., 2018; Li et al., 2016, 2017; Janner et al., 2019; Wu et al., 2015; Mao et al., 2019).”
Model Based Planning with Energy Based Models (2019, methodology): “This means that the dynamics model learned in one domain can then be extracted and applied to other domains and tasks [8], suggesting the better generalizability of model-based methods.”
Entity Abstraction in Visual Model-Based Reinforcement Learning (2019, background): “Those that do not assume such additional information often factorize the entire scene into pixel-level entities [39–41], which do not model objects as coherent wholes.”
Model-Augmented Actor-Critic: Backpropagating through Paths (2020, methodology): “The model can also be used to improve the target value estimates (Feinberg et al., 2018) or to provide additional context to a policy (Du & Narasimhan, 2019).”