Projection-Based Constrained Policy Optimization

By Karthik Narasimhan. International Conference on Learning Representations, 2020

322 citations recorded by the source corpus.

In this paper, we consider the problem of learning control policies that optimize areward function while satisfying constraints due to considerations of safety, fairness, or other costs. We propose a new algorithm - Projection Based ConstrainedPolicy Optimization (PCPO), an iterative method for optimizing policies in a two-step process - the first step performs an unconstrained update while the secondstep reconciles the constraint violation by projection the policy back onto the constraint set. We theoretically analyze PCPO and provide a lower bound on rewardimprovement, as well as an upper bound on constraint violation for each policy update. We further characterize the convergence of PCPO with projection basedon two different metrics - L2 norm and Kullback-Leibler divergence. Our empirical results over several control tasks demonstrate that our algorithm achievessuperior performance, averaging more than 3.5 times less constraint violation andaround 15% higher reward compared to state-of-the-art methods.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/9ec0bd60df62b997bb371e65d254434f8851ea9b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pieter Abbeel
2. Sergey Levine
3. Jun Wang
4. Adam Stooke
5. Joshua Achiam
6. Yaodong Yang
7. Yali Du
8. Alois C. Knoll

In Management Science and Operations Research

1. Pieter Abbeel
2. Joshua Achiam
3. Yaodong Yang
4. Shangding Gu
5. Yali Du
6. Alois C. Knoll
7. Guang Chen
8. Florian Walter

Citations in the corpus

• A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “Model-Free Safe Reinforcement Learning Model-free safe reinforcement learning algorithms, including IPO [166], Lyapunov-Based Safe RL [56, 57], PCPO [307], SAILR [286], SPRL [257], SNO-MDP [284], FOCOPS [316], A-CRL [48] and DCRL [222] all lack convergence rate analysis.”
• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods (2020, background): “Most recently, Yang et al. (2020) extended CPO with a two-step projection-based optimization approach.”
• A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “…over the last few decades, such as policy optimization methods [3], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], formal methods [27], [28], [29], [30], [31], [32], controy theory methods [33], [34], [35], [36], [37], Gaussian processes methods [38], [39],…”