PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos

By Paavo Parmas, Carl E. Rasmussen, Jan Peters, Kenji Doya. 2019

102 citations recorded by the source corpus.

©35th International Conference on Machine Learning, ICML 2018.All Rights Reserved. Previously, the exploding gradient problem has been explained to be central in deep learning and model-based reinforcement learning, because it causes numerical issues and instability in optimization. Our experiments in model-based reinforcement learning imply that the problem is not just a numerical issue, but it may be caused by a fundamental chaos-like nature of long chains of nonlinear computations. Not only do the magnitudes of the gradients become large, the direction of the gradients becomes essentially random. We show that reparameterization gradients suffer from the problem, while likelihood ratio gradients are robust. Using our insights, we develop a model-based policy search framework, Probabilistic Inference for Particle-Based Policy Search (PIPPS), which is easily extensible, and allows for almost arbitrary models and policies, while simultaneously matching the performance of previous data-efficient learning algorithms. Finally, we invent the total propagation algorithm, which efficiently computes a union over all pathwise derivative depths during a single backwards pass, automatically giving greater weight to estimators with lower variance, sometimes improving over reparameterization gradients by 106 times.

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “…Q-learning and one-step predictions via the model from previously seen states (Sutton, 1991), while a variety of recently proposed algorithms employ synthetic model-based rollouts with policy gradients (Parmas et al., 2019; Kaiser et al., 2019a) and actor-critic algorithms (Janner et al., 2019).”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, neutral): “cy learning did not yield an effective algorithm by directly propagating gradients through our uncertainty-aware models. We believe this may be due to chaotic policy gradients, whose recent analysis [Parmas et al., 2018] could help yield a policy-based PETS in future work. Finally, the observation t”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “…as games (Silver et al., 2016; Tesauro, 1995; Brown & Sandholm, 2019) and robotics (Levine et al., 2016; Dasari et al., 2019; Peters et al., 2010; Parmas et al., 2019; Pinto & Gupta, 2016; Nachum et al., 2019a), where the agent can act in the environment and sample as many transitions and…”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “Moreover, prior work commonly resorts to derivative-free optimization for robustness to model errors (Ebert et al., 2017; Chua et al., 2018; Parmas et al., 2019), rather than leveraging analytic gradients offered by neural network dynamics (Henaff et al., 2019; Srinivas et al., 2018).”
Understanding and correcting pathologies in the training of learned optimizers (2018, methodology): “This not only yields computational savings, at the cost of increased bias (Tallec & Ollivier, 2017), but also limits exploding gradients which emerge from too many iterated non-linear function applications (Pascanu et al., 2013; Parmas et al., 2018).”
Offline Policy Selection under Uncertainty (2020, methodology): “While model-based Bayesian approaches exist ( e.g. , (Deisenroth and Rasmussen, 2011) and variants (Parmas et al., 2018)), they typically suﬀer from compounding error, so a model-free approach is preferable.”
Monte Carlo Gradient Estimation in Machine Learning (2019, methodology): “Other generalisations like that of Ruiz et al. (2016), Naesseth et al. (2017) and Parmas et al. (2018), combine the score function method with the pathwise methods; we come back to these hybrid methods in Section 8.”
Gradients are Not All You Need (2021, methodology): “In the context of continuous control this has been explored in depth as a solution to high variance gradients arising from chaos in Parmas et al. [2018], and in the context of learned optimizers in Metz et al. [2019].”