Constrained Policy Optimization

By Joshua Achiam, David Held, Aviv Tamar, Pieter Abbeel. 2017

1,938 citations recorded by the source corpus.

For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (Mnih et al., 2016, Schulman et al., 2015, Lillicrap et al., 2016, Levine et al., 2016) have enabled new capabilities in high-dimensional control, but do not consider the constrained setting. We propose Constrained Policy Optimization (CPO), the first general-purpose policy search algorithm for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration. Our method allows us to train neural network policies for high-dimensional control while making guarantees about policy behavior all throughout training. Our guarantees are based on a new theoretical result, which is of independent interest: we prove a bound relating the expected returns of two policies to an average divergence between them. We demonstrate the effectiveness of our approach on simulated robot locomotion tasks where the agent must satisfy constraints motivated by safety.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Artificial Intelligence, Values, and Alignment (2020, background): “3 Perhaps the most promising approach to this challenge, which involves constrained optimization, come with the strong requirement that these constraints can be fully specified a priori and tends to respect them only ‘in approximation’ (Achiam et al. 2017). Independently, Arnold et al. (2017) have e”
Scalable agent alignment via reward modeling: a research direction (2018, methodology): “The same techniques described here for training a reward model should apply to train a model that estimates side-constraints and blocks low-level actions (Saunders et al., 2018) or enforces constraints during policy updates (Achiam et al., 2017).”
Adversarial Training for High-Stakes Reliability (2022, background): “Certification and verified models Like the subfield of certified or verified robust machine learning [46, 47, 48, 21], we attempt to obtain extreme robustness for relatively easy problems.”
Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “5 and the analysis of CPO [1], we show that CQL provides a ζ-safe policy improvement over π̂β .”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “Our work connects to policy methods with conservative updates (Kakade & Langford, 2002), such as trust region (Schulman et al., 2015; Achiam et al., 2017; Pham et al., 2018) and informationtheoretic methods (Peters & Mülling, 2010; Van Hoof et al.”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “Some of these tools may be of independent interests and used for better analysis of model-free reinforcement learning algorithms such as TRPO Schulman et al. (2015a), PPO Schulman et al. (2017) and CPO Achiam et al. (2017).”
Safe Model-based Reinforcement Learning with Stability Guarantees (2017, background): “In this setting, [14] introduces safety guarantees in terms of constraint satisfaction that hold in expectation.”
Offline RL Without Off-Policy Evaluation (2021, methodology): “For any two policies π and β , Then, Corollary 1 from Achiam et al. [2017] (reproduced below) gives a guarantee for the one-step algorithm.”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, positive): “Following similar derivations in [2, 30], we express the performance difference in linear operator notation: where Π 1 , Π 2 are linear operators S → S × A such that Π i ν ( s, a ) = π i ( a | s ) ν ( s ) .”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “Following the derivations in Achiam et al. (2017) and Nachum et al. (2018a), we express the performance of a policy π in M in terms of linear opterators: where R is a vector representation of the rewards of M , Π π is a linear operator mapping state distributions to state-action distributions…”
Challenges of Real-World Reinforcement Learning (2019, background): “Recent work in RL safety (Dalal et al., 2018; Achiam et al., 2017) has cast safety in the context of Constrained MDPs (CMDPs) (Altman, 1999), and we will concentrate on pre-deﬁned constraints on the environment in this context.”
Batch Policy Learning under Constraints (2019, background): “1 • Conservative policy improvement (Levine & Abbeel, 2014; Schulman et al., 2015; Achiam et al., 2017) is equivalent to G(π) = DKL(π, πk), where πk is some “well-behaving” policy.”
AI Safety Gridworlds (2017, background): “The side constraint could also be directly included in the policy optimization algorithm (Thomas et al., 2015; Achiam et al., 2017).”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, positive): “Formally, To simplify (6) further, we make use of the following property, Property 1 (Achiam et al. [2]) .”
Learning to Walk via Deep Reinforcement Learning (2018, methodology): “Our formulation is an instance of constrained MDP, which has been studied recently in [3, 6, 47].”
Recent advances in reinforcement learning in finance (2021, background): “Another line of work focuses on constrained RL problemswith different risk criteria (Achiam et al., 2017; Chow et al., 2017, 2015; Ding et al., 2021; Tamar et al., 2015; Zheng &Ratliff, 2020).”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “Recent work in RL safety (Dalal et al. 2018; Achiam et al. 2017; Tessler et al. 2018; Satija et al. 2020) has cast safety in the context of Constrained MDPs (CMDPs) (Altman 1999), and we will concentrate on pre-defined constraints on the environment in this context.”
Projection-Based Constrained Policy Optimization (2020, methodology): “Then perform policy updates using a conditional gradient descent with line search to ensure constraint satisfaction (Achiam et al., 2017).”