High-Dimensional Continuous Control Using Generalized Advantage Estimation
By John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, Pieter Abbeel. 2015
4,662 citations recorded by the source corpus.
Policy gradient methods are an appealing approach in reinforcement learning because they directly optimize the cumulative reward and can straightforwardly be used with nonlinear function approximators such as neural networks. The two main challenges are the large number of samples typically required, and the difficulty of obtaining stable and steady improvement despite the nonstationarity of the incoming data. We address the first challenge by using value functions to substantially reduce the variance of policy gradient estimates at the cost of some bias, with an exponentially-weighted estimator of the advantage function that is analogous to TD(lambda). We address the second challenge by using trust region optimization procedure for both the policy and the value function, which are represented by neural networks. Our approach yields strong empirical results on highly challenging 3D locomotion tasks, learning running gaits for bipedal and quadrupedal simulated robots, and learning a policy for getting the biped to stand up from starting out lying on the ground. In contrast to a body of prior work that uses hand-crafted policy representations, our neural network policies map directly from raw kinematics to joint torques. Our algorithm is fully model-free, and the amount of simulated experience required for the learning tasks on 3D bipeds corresponds to 1-2 weeks of real time.
Subfields: Artificial Intelligence, Biomedical Engineering, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Mohit Bansal
- Jianfeng Gao
- Ruslan Salakhutdinov
- Quoc V. Le
- Percy Liang
In Biomedical Engineering
- Sergey Levine
- Pieter Abbeel
- Vikash Kumar
- Chelsea Finn
- Igor Mordatch
- Jitendra Malik
- Vladlen Koltun
- Marco Hutter
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Shimon Whiteson
- Ofir Nachum
- Shixiang Shane Gu
- Yuxi Li
- Abhishek Gupta
Citations in the corpus
- RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, methodology): “We optionally apply Generalized Advantage Estimation (GAE) (Schulman et al., 2016) to further reduce the variance.”
- Learning to summarize from human feedback (2020, methodology): “Table 4 documents the formats we used (with pythonic format strings).”
- Emergence of Locomotion Behaviours in Rich Environments (2017, background): “[10, 7, 11, 12, 6, 13]), although impressive, have focused on comparatively small problems, and the use of large-scale, distributed optimization is less widespread and the corresponding algorithms are less well developed (but see e.”
- Sample Efficient Actor-Critic with Experience Replay (2016, background): “Despite the effectiveness of their TRPO method, it requires repeated computation of Fisher-vector products for each update.”
- Benchmarking Deep Reinforcement Learning for Continuous Control (2016, methodology): “…2015a), Walker (Raibert & Hodgins, 1991; Erez et al., 2011; Levine & Koltun, 2013; Schulman et al., 2015a), Half-Cheetah (Wawrzy ´ nski, 2007; Heess et al., 2015b), Ant (Schulman et al., 2015b), Simple Humanoid (Tassa et al., 2012; Schul-man et al., 2015b), and Full Humanoid (Tassa et al., 2012).”
- Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “By storing the agent’s data in an experience replay memory, the data can be batched (Riedmiller, 2005; Schulman et al., 2015a) or randomly sampled (Mnih et al., 2013; 2015; Van Hasselt et al., 2015) from different time-steps.”
- Memory-based control with recurrent neural networks (2015, background): “Several recent papers successfully apply model-free, direct policy search methods to the problem of learning neural network control policies for challenging continuous domains with many degrees of freedoms [2, 6, 14, 21, 22, 12].”
- Constrained Policy Optimization (2017, methodology): “We use GAE-λ (Schulman et al., 2016) to estimate the advantages and constraint advantages, with neural network value functions.”
- Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “Based on the similarity to policy gradients, we adopt the widely used baseline technique from RL, as previously studied in Williams (1992); Baxter & Bartlett (2001); Schulman et al. (2016) and in the context of DPG in (Korbak et al., 2022b).”
- Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “To increase training stability, advantage is appoximated using Generalized Advantage Estimation (Schulman et al., 2015b).”
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “This allows for learning value functions in infinite-horizon settings, where the backup would otherwise be non-convergent for γ = 1, and reduces variance for Monte Carlo advantage estimators in policy gradient algorithms (Schulman et al., 2016).”
- Concrete Problems in AI Safety (2016, background): “If we specialize to prediction tasks and let x denote the input and y denote the output (prediction target), then one possibility is to make the covariate shift assumption that p 0 ( y | x ) = p ( y | x ).”
- Self-Critical Sequence Training for Image Captioning (2016, background): “The recent discovery that REINFORCE with proper bias correction using learned “baselines” is effective has led to a flurry of work in applying REINFORCE to problems in RL, supervised learning, and variational inference [16, 17, 18].”
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “𝑡 is the advantage, which is computed by applying Generalized Advantage Estimation (GAE) (Schulman et al., 2015 each token (Ouyang et al., 2022), i.e., where 𝑟 𝜑 is the reward model, 𝜋 𝑟𝑒𝑓 is the reference model, which is usually the initial SFT model, and 𝛽 is the coefficient of the KL…”
- Deep Reinforcement Learning that Matters (2017, methodology): “We also include all our experimental results, with baseline algorithms DDPG (Lillicrap et al. 2015b), TRPO (Schulman et al. 2015a), PPO (Schulman et al. 2017) and ACKTR (Wu et al. 2017)) as discussed in the paper.”
- Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, background): “We are aware that TRPO with the addition of GAE and enough fine-tuning can be made to achieve good results on Humanoid (Schulman et al., 2016).”
- Equivalence Between Policy Gradients and Soft Q-Learning (2017, methodology): “We shall consider an entropy-regularized version of the reinforcement learning problem, following various prior work (Ziebart [2010], Fox et al.”
- Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, background): “What is feasible on a simulated system, where hundreds of millions of control steps are possible (Mnih et al., 2016; Schulman et al., 2016), does not necessarily transfer to real robot applications due to unrealistic learning times.”