PILCO: A Model-Based and Data-Efficient Approach to Policy Search

By Marc Peter Deisenroth, Carl E. Rasmussen. 2011

1,810 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Continuous control with deep reinforcement learning (2015, methodology): “However, due to the high computational demand, PILCO is “impractical for high-dimensional problems” [22].”
Learning Continuous Control Policies by Stochastic Value Gradients (2015, methodology): “If a differentiable environment model is available, one can link together the policy, model, and reward function to compute an analytic policy gradient by backpropagation of reward along a trajectory [18, 11, 6, 9].”
End-to-End Training of Deep Visuomotor Policies (2015, methodology): “ We compare to REPS (Peters et al., 2010), reward-weighted regression (RWR) (Peters and Schaal, 2007; Kober and Peters, 2009), the cross-entropy method (CEM) (Rubinstein and Kroese, 2004), and PILCO (Deisenroth and Rasmussen, 2011). We also use iLQG (Li and Todorov, 2004) with a known model as a bas”
From Pixels to Torques: Policy Learning with Deep Dynamical Models (2015, methodology): “These ideas have been successfully applied to RL problems in control and robotics by Atkeson & Schaal (1997), Bagnell & Schneider (2001), Deisenroth & Rasmussen (2011) and Pan & Theodorou (2014), for instance.”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, background): “[8] MP.”
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, background): “Model-based algorithms have explored a variety of dynamics estimation schemes, including Gaussian processes [20], mixture models [21], and local linear system estimation [22], with a more detailed overview in a recent survey [23].”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “In the regime of control tasks with only a few state variables, such as the cart pole and mountain car tasks, PILCO (Deisenroth and Rasmussen, 2011) achieves remarkable sample efﬁciency using Gaussian processes to model the dynamics.”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “The more recent PILCO [10, 53] is a probabilistic model-based search policy method designed to solve difficult control problems.”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, background): “A class of value gradient methods (Deisenroth & Rasmussen, 2011; Fairbank & Alonso, 2012; Heess et al., 2015) also differentiates through models to train a policy.”
Value Iteration Networks (2016, methodology): “Our approach is different from model-based RL [22, 5], which requires system identiﬁcation to map the observations to a dynamics model, which is then solved for a policy.”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…most obvious in model-based RL, where it is used to train predictive models that can then be used for planning or policy learning (Sutton, 1990; Silver et al., 2008; Fairbank, 2008; Deisenroth & Rasmussen, 2011; Lampe & Riedmiller, 2014; Heess et al., 2015b; Janner et al., 2020; Amos et al., 2021).”
Continuous Deep Q-Learning with Model-based Acceleration (2016, methodology): “While prior methods propose a variety of model classes, including neural networks (Heess et al., 2015), Gaussian processes (Deisenroth & Rasmussen, 2011), and locally-weighted regression (Atkeson et al., 1997), we found that we could obtain good results by using iteratively reﬁtted time-varying…”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “In low-dimensional MDPs, such uncertainty estimates can be produced by means of Bayesian models such as Gaussian processes (Deisenroth and Rasmussen, 2011), while for higher-dimensional problems, Bayesian neural networks and bootstrap ensembles can be utilized (Chua et al., 2018; Janner et al.,…”
Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data (2016, background): “Estimating probabilistic models for sequential data is central to many domains, such as audio, natural language or physical plants [7, 19, 3, 4, 13].”
Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, background): “, [17, 18]) do not have U as input and cannot directly produce counterfactual outcomes–counterfactual reasoning is not possible without a causal model.”
Causally Correct Partial Models for Reinforcement Learning (2020, neutral): “…include action-conditional, next-step models (Oh et al., 2015; Ha & Schmidhuber, 2018; Chiappa et al., 2017; Schmidhuber, 2010; Xie et al., 2016; Deisenroth & Rasmussen, 2011; Lin & Mitchell, 1992; Li et al., 2015; Diuk et al., 2008; Igl et al., 2018; Ebert et al., 2018; Kaiser et al., 2019;…”
Shaping Belief States with Generative Environment Models for RL (2019, background): “The idea of learning general world models to support decision-making is probably one of the most pervasive ideas in AI research, [30, 11, 41, 42, 14, 43, 44, 45, 46, 33, 30, 47, 48, 3, 49, 50, 51, 52].”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, background): “Model-based RL The promises of learning and leveraging a model of the environment have been discussed before (Sutton and Barto, 2018), yet doing so in practice for complex environments remains a challenge (Deisenroth and Rasmussen, 2011).”