Trust-PCL: An Off-Policy Trust Region Method for Continuous Control

By Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans. 2017

115 citations recorded by the source corpus.

Trust region methods, such as TRPO, are often used to stabilize policy optimization algorithms in reinforcement learning (RL). While current trust region strategies are effective for continuous control, they typically require a prohibitively large amount of on-policy interaction with the environment. To address this problem, we propose an off-policy trust region method, Trust-PCL. The algorithm is the result of observing that the optimal policy and state values of a maximum reward objective with a relative-entropy regularizer satisfy a set of multi-step pathwise consistencies along any path. Thus, Trust-PCL is able to maintain optimization stability while exploiting off-policy data to improve sample efficiency. When evaluated on a number of continuous control tasks, Trust-PCL improves the solution quality and sample efficiency of TRPO.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “In the case of path consistency methods (Nachum et al., 2017a,b), the maximum entropy framework offers an appealing mechanism for off-policy learning.”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “Prior work has proposed model-free deep RL algorithms that perform on-policy learning with entropy maximization (O’Donoghue et al., 2016), as well as off-policy methods based on soft Q-learning and its variants (Schulman et al., 2017a; Nachum et al., 2017a; Haarnoja et al., 2017).”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “We make the following additional practical modifications: • As entropy regularization has been shown to be important on these tasks (Nachum et al., 2017b; Haarnoja et al., 2018), we augment the rewards with a causal entropy term; i.e., replace r in (25) with r − τ log π(a′|s′), where τ is learned…”
Behavior Regularized Offline Reinforcement Learning (2019, methodology): “value function (Q-value function) learned with function approximators, as suggested by Sutton (1995); Baird (1995); Tsitsiklis & Van Roy (1997); Van Hasselt et al.”
Generalized Off-Policy Actor-Critic (2019, background): “The target policy can then be derived from the value function directly (O’Donoghue et al., 2016; Nachum et al., 2017a; Schulman et al., 2017a).”
Soft Actor-Critic Algorithms and Applications (2018, methodology): “Prior work has proposed model-free deep RL algorithms that perform on-policy learning with entropy maximization (O’Donoghue et al., 2016), as well as off-policy methods based on soft Q-learning and its variants (Schulman et al., 2017a; Nachum et al., 2017a; Haarnoja et al., 2017).”
Data-Efficient Hierarchical Reinforcement Learning (2018, methodology): “In this way, we are able to use past experience for training the higher-level policy, taking advantage of progress made in recent years to provide stable, robust, and general off-policy RL methods [12, 28, 3].”
A Kernel Loss for Solving the Bellman Equation (2019, methodology): “Here, we take the first approach, leveraging our method to improve the critic update step in Trust-PCL (Nachum et al., 2018).”
Text Generation with Efficient (Soft) Q-Learning (2021, background): “Previous work in robotic and game control demonstrated that this formulation provides a substantial improvement in exploration and robustness [70, 57, 58, 69, 50, 41, 17, 36, 53, 37, 13].”
Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods (2018, methodology): “In our experiments, we use a variant Trust-PCL [30], which uses a Gaussian policy and modifies the entropy regularizer to relative entropy with respect to a prior version of the policy.”
Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past (2019, background): “…tasks, outperforming prior on-policy and off-policy methods, including proximal policy optimization (PPO) [29], deep determinisitc policy gradient (DDPG) [18], soft Q-learning [9], twin delayed determinisitc policy gradient (TD3) [7], and trust region path consistency learning (Trust-PCL) [24].”
Learning to Walk via Deep Reinforcement Learning (2018, background): “In practice, we parameterize a Gaussian policy with parameters , and learn them using stochastic gradient descent for the discounted, infinite horizon problem.”
Dealing with Non-Stationarity in MARL via Trust-Region Decomposition (2021, methodology): “Nachum et al. (2018) proposed an off-policy trust-region method, Trust-PCL, which introduced relative entropy regularization to maintain optimization stability while exploiting off-policy data.”
SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation (2017, background): “That said, Nachum et al. (2017, 2018) also exploit the consistency condition in Theorem 3 and propose the PCL algorithm which optimizes the upper bound of the mean squared consistency Bellman error (7).”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, result): “Other works in this line of research include Nachum et al. (2017a); O’Donoghue et al. (2016); Schulman et al. (2017); Nachum et al. (2017b); Haarnoja et al. (2017; 2018), which are mainly developed in a tabular setting and do not have a convergence analysis under function approximation.”
Composable Deep Reinforcement Learning for Robotic Manipulation (2018, methodology): “[30] approximates the maximum entropy distribution with a Gaussian, and, to our knowledge, soft Q-learning [7] is the only method that approximates the true maximum entropy policy with an expressive inference network, which is essential for tasks requiring diverse behaviour at test time and involvin”
Provably Convergent Off-Policy Actor-Critic with Function Approximation (2019, background): “Another line of policy-based off-policy algorithms involves reward shaping via policy entropy (Nachum et al., 2017a; O’Donoghue et al., 2016; Schulman et al., 2017; Nachum et al., 2017b; Haarnoja et al., 2017; 2018; Dai et al., 2017) and is orthogonal to our work.”