Proximal Policy Optimization Algorithms
By John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. arXiv.org, 2017
29,461 citations recorded by the source corpus.
We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
Subfields: Artificial Intelligence, Control and Systems Engineering, Management Science and Operations Research, Reinforcement Learning, Optimization & Training
View Paper: https://arxiv.org/abs/1707.06347
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Pieter Abbeel
- Jianfeng Gao
- Percy Liang
- Ruslan Salakhutdinov
In Control and Systems Engineering
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Vikash Kumar
- Abhishek Gupta
- Abhinav Kumar Gupta
- Jan Peters
- Shixiang Shane Gu
In Management Science and Operations Research
- Sergey Levine
- Pieter Abbeel
- Ofir Nachum
- Shimon Whiteson
- R´emi Munos
- Lihong Li
- George Tucker
- Nan Jiang
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shimon Whiteson
- Shixiang Shane Gu
- Yuxi Li
- Abhishek Gupta
In Optimization & Training
- Yoshua Bengio
- Sergey Levine
- Quoc V. Le
- Pieter Abbeel
- Frank Hutter
- Simon Lacoste-Julien
- Tengyu Ma
- Michael I. Jordan
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “For each PPO iteration we use a batch size of 512, a PPO clip threshold of 0.2, a mini-batch size of 64, and take one gradient step per mini-batch.”
- Training language models to follow instructions with human feedback (2022, methodology): “Again following Stiennon et al. (2020), we fine-tuned the SFT model using PPO (Schulman et al., 2017).”
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “…for compatibility with the dataset of preferences under a preference model such as the Bradley-Terry model [5], then fine-tune a language model to maximize the given reward using reinforcement learning algorithms, commonly REINFORCE [44], proximal policy optimization (PPO; [37]), or variants [32].”
- Fine-Tuning Language Models from Human Preferences (2019, methodology): “Train π via Proximal Policy Optimization (PPO, Schulman et al. (2017)) with reward R from (2) on x ∼ D.”
- An Empirical Model of Large-Batch Training (2018, methodology): “• Dota The OpenAI Dota team has made it possible to train PPO [SWD + 17] agents on both Dota 1v1 and 5v5 environments (the latter being preliminary ongoing work).”
- OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, methodology): “This is achieved using the PPO algorithm [15].”
- Augmented Language Models: a Survey (2023, methodology): “Optimization is done through policy gradient algorithms like REINFORCE (Williams, 1992), PPO and similar variants (Schulman et al., 2017; Ramamurthy et al., 2022).”
- WebGPT: Browser-assisted question-answering with human feedback (2021, methodology): “• We used the same number of parallel environments and timesteps per rollout as the default for PPO, even though it resulted in slow PPO iterations (lasting multiple hours).”
- Learning to summarize from human feedback (2020, methodology): “We primarily do this using reinforcement learning, by treating the output of the reward model as a reward for the entire summary that we maximize with the PPO algorithm [58], where each time step is a BPE token.”
- ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “Cautious RL Standard works in RL proposed ways to avoid catastrophic moves of the policy parameters [36, 37], by enforcing the new policy to stay Importance Sampling for Reinforcement Learning In RL, IS is generally used for sample ef-ficiency purposes: in off-policy policy gradient methods, IS…”
- Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “Setup: We use the setup from [59] who train an agent using reinforcement learning (PPO [60]) in the Gibson environment [73].”
- Scalable agent alignment via reward modeling: a research direction (2018, background): “Major improvements have originated from scaling deep RL to a distributed setting across many machines (Mnih et al., 2016; Schulman et al., 2017; Barth-Maron et al., 2018; Horgan et al., 2018; Espeholt et al., 2018; Anonymous, 2019a).”
- Scaling Laws for Reward Model Overoptimization (2022, methodology): “The RL experiments use Proximal Policy Optimization (PPO) [Schulman et al., 2017].”
- Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “The overall objective with the KL-penalty is maximized using an RL algorithm of choice including: PPO (Schulman et al., 2017) as in Ziegler et al. (2019) or Q-learning (Mnih et al., 2013) as in Jaques et al. (2017).”
- Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “, PPO (Schulman et al., 2017)), based on both automatic and human evaluations.”
- On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, methodology): “…candidates include further reducing the variance using a learned value function (Konda & Tsitsiklis, 2000) and preventing detrimentally large policy updates by maintaining a trust region in the policy space – akin to techniques such as TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017b).”
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “We compare AWR to a number of state-of-the-art RL algorithms, including on-policy algorithms, such as TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017), off-policy algorithms, such as DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018), and SAC (Haarnoja et al., 2018a), as well as…”
- STaR: Bootstrapping Reasoning With Reasoning (2022, methodology): “Thus, STaR approximates J by (1) greedily decoding samples of (r̂i, ŷi) to reduce variance of this estimate (at the cost of potentially biased exploration of rationales), and (2) taking multiple gradient steps on the same batch of data (similar to some policy gradient algorithms [28]).”