Proximal Policy Optimization Algorithms

By John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. arXiv.org, 2017

29,461 citations recorded by the source corpus.

We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.

Subfields: Artificial Intelligence, Control and Systems Engineering, Management Science and Operations Research, Reinforcement Learning, Optimization & Training

View Paper: https://arxiv.org/abs/1707.06347

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Mohit Bansal
  4. Quoc V. Le
  5. Pieter Abbeel
  6. Jianfeng Gao
  7. Percy Liang
  8. Ruslan Salakhutdinov

In Control and Systems Engineering

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Vikash Kumar
  5. Abhishek Gupta
  6. Abhinav Kumar Gupta
  7. Jan Peters
  8. Shixiang Shane Gu

In Management Science and Operations Research

  1. Sergey Levine
  2. Pieter Abbeel
  3. Ofir Nachum
  4. Shimon Whiteson
  5. R´emi Munos
  6. Lihong Li
  7. George Tucker
  8. Nan Jiang

In Reinforcement Learning

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Ofir Nachum
  5. Shimon Whiteson
  6. Shixiang Shane Gu
  7. Yuxi Li
  8. Abhishek Gupta

In Optimization & Training

  1. Yoshua Bengio
  2. Sergey Levine
  3. Quoc V. Le
  4. Pieter Abbeel
  5. Frank Hutter
  6. Simon Lacoste-Julien
  7. Tengyu Ma
  8. Michael I. Jordan

Citations in the corpus