Trust Region Policy Optimization
By John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, Philipp Moritz. International Conference on Machine Learning, 2015
8,139 citations recorded by the source corpus.
In this article, we describe a method for optimizing control policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified scheme, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.
Subfields: Artificial Intelligence, Reinforcement Learning, Optimization & Training
View Paper: https://arxiv.org/abs/1502.05477
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Ruslan Salakhutdinov
- Mohit Bansal
- Chelsea Finn
- Quoc V. Le
- Jianfeng Gao
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shimon Whiteson
- Shixiang Shane Gu
- Yuxi Li
- Aviral Kumar
In Optimization & Training
- Sergey Levine
- Yoshua Bengio
- Quoc V. Le
- Pieter Abbeel
- Michael I. Jordan
- Simon Lacoste-Julien
- Peter Richtárik
- Jason D. Lee
Citations in the corpus
- Continuous control with deep reinforcement learning (2015, background): “Another approach, trust region policy optimization (TRPO) [20], directly constructs stochastic neural network policies without decomposing problems into optimal control and supervised phases.”
- Domain randomization for transferring deep neural networks from simulation to the real world (2017, neutral): “Learning in simulation is especially promising for building on recent results using deep reinforcement learning to achieve human-level performance on tasks like Atari [29] and robotic control [26], [41].”
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, methodology): “The gradient updates are computed using vanilla policy gradient (REINFORCE) (Williams, 1992), and we use trust-region policy optimization (TRPO) as the meta-optimizer (Schulman et al., 2015).”
- RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, methodology): “These methods can learn a task using a few minutes to a few hours of real experience, compared to days or even weeks required by previous methods (Schulman et al., 2015; 2016; Lilli-crap et al., 2015).”
- ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “Cautious RL Standard works in RL proposed ways to avoid catastrophic moves of the policy parameters [36, 37], by enforcing the new policy to stay Importance Sampling for Reinforcement Learning In RL, IS is generally used for sample ef-ficiency purposes: in off-policy policy gradient methods, IS…”
- Deep Reinforcement Learning from Human Preferences (2017, methodology): “For the simulated robotics tasks, we optimize policies using trust region policy optimization (TRPO, Schulman et al., 2015) with discount rate γ = 0 .”
- Emergence of Locomotion Behaviours in Rich Environments (2017, methodology): “A popular algorithm that makes use of this idea is trust region policy optimization (TRPO; [7]).”
- OpenAI Gym (2016, background): “…that combine deep learning with reinforcement learning have led to a great deal of excitement in the field, as it has become evident that general algorithms such as policy gradients and Q-learning can achieve good performance on difficult problems, without problem-specific engineering [2, 3, 4].”
- Benchmarking Deep Reinforcement Learning for Continuous Control (2016, methodology): “Relative Entropy Policy Search (REPS) (Peters et al., 2010): This algorithm limits the loss of information per iteration and aims to ensure a smooth learning progress (Deisenroth et al., 2013).”
- Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “By storing the agent’s data in an experience replay memory, the data can be batched (Riedmiller, 2005; Schulman et al., 2015a) or randomly sampled (Mnih et al., 2013; 2015; Van Hasselt et al., 2015) from different time-steps.”
- High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, methodology): “However, it uses the methods specified in [12] for optimization and stepsize determination, and it uses  in the gradient estimator.”
- Memory-based control with recurrent neural networks (2015, background): “Several recent papers successfully apply model-free, direct policy search methods to the problem of learning neural network control policies for challenging continuous domains with many degrees of freedoms [2, 6, 14, 21, 22, 12].”
- Learning Continuous Control Policies by Stochastic Value Gradients (2015, background): “Walker is a planar biped, based on the environment from [22].”
- One-Shot Imitation Learning (2017, background): “…including Backgammon (Tesauro, 1995), helicopter control (Ng et al., 2003), Atari (Mnih et al., 2015), Go (Silver et al., 2016), continuous control in simulation (Schulman et al., 2015; Heess et al., 2015; Lillicrap et al., 2015) and on real robots (Peters & Schaal, 2008; Levine et al., 2016).”
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, methodology): “Well-known examples include Trust Region Policy Optimization (TRPO) [50], and use conservative, KL-regularized policy updates to restrict the RL algorithm to stay close to its own prior policy (e.”
- Scalable agent alignment via reward modeling: a research direction (2018, background): “In recent years the machine learning community has made great strides in designing more and more capable deep reinforcement learning algorithms, both value-based methods derived from Q-learning (Mnih et al., 2015) and policy-gradient methods (Schulman et al., 2015; Lillicrap et al., 2015).”
- Constrained Policy Optimization (2017, methodology): “Like (Schulman et al., 2015), we approximately compute them using the conjugate gradient method.”
- Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “To increase training stability, advantage is appoximated using Generalized Advantage Estimation (Schulman et al., 2015b).”