V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control

By H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W. Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala and 4 more. 2019

141 citations recorded by the source corpus.

Some of the most successful applications of deep reinforcement learning to challenging domains in discrete and continuous control have used policy gradient methods in the on-policy setting. However, policy gradients can suffer from large variance that may limit performance, and in practice require carefully tuned entropy regularization to prevent policy collapse. As an alternative to policy gradient algorithms, we introduce V-MPO, an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO) that performs policy iteration based on a learned state-value function. We show that V-MPO surpasses previously reported scores for both the Atari-57 and DMLab-30 benchmark suites in the multi-task setting, and does so reliably without importance weighting, entropy regularization, or population-based tuning of hyperparameters. On individual DMLab and Atari levels, the proposed algorithm can achieve scores that are substantially higher than has previously been reported. V-MPO is also applicable to problems with high-dimensional, continuous action spaces, which we demonstrate in the context of learning to control simulated humanoids with 22 degrees of freedom from full state observations and 56 degrees of freedom from pixel observations, as well as example OpenAI Gym tasks where V-MPO achieves substantially higher asymptotic scores than previously reported.

Subfields: Artificial Intelligence, Cell Biology, Molecular Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cell Biology

In Molecular Biology

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “…with respect to π is where the softmax function is deﬁned as Thus, we see that the use of KL-divergence leads to a dual formulation which bears similarities to max-likelihood policy learning, a common goal of a number of recent works (Abdolmaleki et al., 2018; Song et al., 2019; Peng et al., 2019).”
What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, methodology): “D Experiment Policy Losses D.1 Design For each of the 5 environments, we sampled 2000 choice configurations where we sampled the following choices independently and uniformly from the following ranges: • num_epochs (C3): {1, 3, 10} • Policy loss (C14): {AWR, PG, PPO, RPA, V-MPO, V-Trace} –”
Stabilizing Transformers for Reinforcement Learning (2019, methodology): “V-MPO was shown to achieve state-of-the-art results for LSTM-based agents on the multi-task DMLab-30 benchmark suite.”
A Generalist Agent (2022, background): “Several poplar RL agents achieve good multi-task RL results within single domains such as Atari57 and DMLab (Espeholt et al., 2018; Song et al., 2020; Hessel et al., 2019).”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “Additionally, the learning rate and V-MPO hyperparameter 𝜖𝛼 are added to the set of hyperparameters modified by PBT. PBT requires a fitness function to compare two agents and propagate the preferred agent.”
Catch & Carry (2019, methodology): “Policy updates are perfomed using a version of V-MPO [Song et al. 2020], which is in the MPO family of approaches [Abdolmaleki et al. 2018]; however, while V-MPO proposes the use of on-policy updates, we perform updates from a replay buffer, which we found stabilized learning in our setting.”
Physically Embedded Planning Problems: New Challenges for Reinforcement Learning (2020, methodology): “RL methods have also been able to tackle domains where perception is a challenge, due to partial observability or a complex sensory stream (Beattie et al., 2016; Gregor et al., 2019; Paine et al., 2020; Song et al., 2020; Wayne et al., 2018).”
What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study (2021, background): “…(KL) between a reference N (0 , 1) action distribution and the current policy ( C37 , C43 ) and the KL divergence and reverse KL divergence between the current policy and the behavioral one ( C35 , C41 , C36 , C42 ), as well as the “decoupled” KL divergence from [18, 34] ( C38 , C39 , C44 , C45 ).”
Revisiting Design Choices in Proximal Policy Optimization (2020, unknown): “Finally, we believe that our insights on PPO can also help to better understand other policy gradient algorithms such as Natural Policy Gradient (NPG) [13] and V-MPO [23], as they are closely connected to KL-regularized PPO as discussed in Appendix A.”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, methodology): “We center our analyses around MPO, AWR, and SAC, because they are representative algorithms that span both EM-based [49, 50, 43, 44, 1, 56, 42] and KL-control-based [59, 51, 12, 29, 21, 33, 32] RL and achieve some of the most competitive performances on popular benchmarks [6, 58].”
Reinforced Self-Training (ReST) for Language Modeling (2023, methodology): “The main difference is that we use a state-value function instead of Q funtion with v-trace (Espeholt et al., 2018), similarly to V-MPO (Song et al., 2020).”
Intra-agent speech permits zero-shot task acquisition (2022, methodology): “We used V-MPO [16] to optimize q ω ( y | x ) .”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, methodology): “There are also some previous works using softmax to deal with the partition function, such as AWAC [29]) and VMPO [45], which has been conﬁrmed to promote performance improvement.”
Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning (2020, methodology): “19). Predictive models for representation learning have been widely used as auxiliary tasks for deep RL (Jaderberg et al.,2017;Oord et al.,2018;Zhang et al., 2019;Aytar et al.,2018;Gregor et al.,2019;Song et al., 2020). However, to learn a rich and useful representation, these methods may demand a h”
A data-driven approach for learning to control computers (2022, methodology): “…mixture of imitation learning (behavioural cloning (BC) (Pomerleau, 1989)) and reinforcement learning (RL) (using the VMPO algo-rithm (Song et al., 2019)), which is a combination that has been largely successful in many challenging domains (Silver et al., 2016; Vinyals et al., 2019;…”
Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation (2021, methodology): “For all models, we sweep over the V-MPO target network update frequency K L ∈ { 1 , 10 , 100 } .”
A Distributional View on Multi-Objective Policy Optimization (2020, methodology): “We combine it with maximum a posteriori policy optimization (MPO) (Abdolmaleki et al., 2018a;b), an off-policy actor-critic RL algorithm, and V-MPO (Song et al., 2020), an on-policy variant of MPO.”
RL Unplugged: A Suite of Benchmarks for Ofﬂine Reinforcement Learning (2020, methodology): “These tasks are made up of the corridor locomotion tasks involving the CMU Humanoid [Tassa et al., 2020], for which prior efforts have either used motion capture data [Merel et al., 2019a,b] or training from scratch [Song et al., 2020].”