Maximum a Posteriori Policy Optimisation

By Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, R´emi Munos, Nicolas Manfred Otto Heess, Martin A. Riedmiller. 2018

593 citations recorded by the source corpus.

We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are competitive with the state-of-the-art in deep reinforcement learning. In particular, for continuous control, our method outperforms existing methods with respect to sample efficiency, premature convergence and robustness to hyperparameter settings while achieving similar or better final performance.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, background): “π∗ Ψ(at|st) = exp(Ψ(st, at)) (9) Because it avoids taking a hard max over noisy estimates, Ψ-learning leads to less overestimation of future reward [1, 21].”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “The approach we propose is based on KL-control, a branch of stochastic optimal control (SOC) (Sten-gel, 1986) where the Kullback-Leibler (KL) divergence from some distribution is used to regularize an RL policy (Abdolmaleki et al., 2018; Kappen et al., 2012; Rawlik et al., 2012; Todorov, 2007).”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “This objective differs from the ones used in the derivations of related algorithms, such as RWR and REPS (Peters & Schaal, 2007; Peters et al., 2010; Abdolmaleki et al., 2018), which maximize the expected return J ( π ) instead of the expected improvement.”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “…r(τ) is positive and bounded and applying Jensen’s inequality results in a variety of algorithms (Peters and Schaal, 2007; Hachiya et al., 2009; Neumann, 2011; Abdolmaleki et al., 2018), including reward-weighted regression (Peters and Schaal, 2007), that all follow the following general…”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “We systematically analyze the challenges with using standard off-policy RL algorithms [7, 9, 1] for this problem, and introduce a simple actor critic algorithm that elegantly bridges data-driven pre-training from offline data and improvement with online data collection.”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, methodology): “We compare our agent to two baselines: i) MPO raw : Maximum a Posteriori Policy Optimization (MPO) (Abdolmaleki et al., 2018), a state-of-the-art model-free continuous control algorithm known for its data-eﬃciency. ii) MPO handheld : MPO endowed with our agent’s vision module and exploration policy.”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, methodology): “…also bears similarity to previous works that use (relative) entropy regularized policy updates which implement constraints with respect to either a fixed (e.g. uniform) policy (e.g. Haarnoja et al., 2018a) or, in a trust-region like scheme, to the previous policy (e.g. Abdolmaleki et al., 2018).”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “…with respect to π is where the softmax function is deﬁned as Thus, we see that the use of KL-divergence leads to a dual formulation which bears similarities to max-likelihood policy learning, a common goal of a number of recent works (Abdolmaleki et al., 2018; Song et al., 2019; Peng et al., 2019).”
Behavior Regularized Offline Reinforcement Learning (2019, background): “Abdolmaleki et al. (2018); Kakade (2002); Peters et al. (2010); Schulman et al. (2015); Nachum et al. (2017). Notably, Jaques et al. (2019) apply this idea to offline RL in discrete action domains by introducing a KL value penalty in the Q-value definition.”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, background): “These approaches are very closely related to maximum entropy reinforcement learning techniques which add an entropy regularization ‘bonus’ to prevent early convergence to deterministic policies and thereby heuristically encourage exploration [57, 59, 28, 35, 2, 25].”
Critic Regularized Regression (2020, methodology): “Furthermore, the policy update in MPO is ineffective at reducing the stochasticity of the prior policy which can lower performance in humanoid and manipulation environments.”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, methodology): “We instead choose to maximize the log of the return, and then introduce a variational trajectory distribution q ( τ ) and a corresponding variational lower bound [3] In our setting of intervention-based learning, we view q ( τ ) as a dataset distribution that is curated by the human.”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, methodology): “, 2020) which uses advantage weighted regression to learn a prior policy to which the policy is constrained to stay close to via MPO (Abdolmaleki et al., 2018).”
What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, background): “In particular, we consider the following regularization terms: entropy (C40, C46), the Kullback–Leibler divergence (KL) between a reference N (0, 1) action distribution and the current policy (C37, C43) and the KL divergence and reverse KL divergence between the current policy and the behavioral one”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “Introduced by Abdolmaleki et al. (2018), the MPO algorithm takes a two-pronged approach to policy optimization, inspired by the classical expectation-maximization (EM) algorithm.”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “In contrast, the model-free methods either learn the latent representation indirectly by optimizing the RL objective (Barth-Maron et al., 2018; Abdolmaleki et al., 2018) or by employing auxiliary losses that provide additional supervision (Yarats et al.”
Soft Actor-Critic Algorithms and Applications (2018, methodology): “Similar approach was taken in (Abdolmaleki et al., 2018), where the policy was constrained to remain close to the previous policy.”
Stabilizing Transformers for Reinforcement Learning (2019, methodology): “V-MPO was shown to achieve state-of-the-art results for LSTM-based agents on the multi-task DMLab-30 benchmark suite.”
A Generalist Agent (2022, methodology): “The training procedure was to train single-task MPO (Abdolmaleki et al., 2018) experts on each of the MT-50 tasks individually, recording the trajectories produced while training.”
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (2019, methodology): “14 as in Abdolmaleki et al. (2018b); the precise form is given in Appendix C.”