Reward Constrained Policy Optimization

By Chen Tessler, Daniel Jaymin Mankowitz, Shie Mannor. International Conference on Learning Representations, 2018

717 citations recorded by the source corpus.

Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.

Subfields: Artificial Intelligence, Computer Networks and Communications, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/cb7c479a36520da1caeeec67db10772351a390c6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Management Science and Operations Research

Citations in the corpus

Responsive Safety in Reinforcement Learning by PID Lagrangian Methods (2020, methodology): “Adaptations of the Lagrange multiplier method to the actor-critic RL setting have been shown to converge to the optimal, constraint-satisfying solution under certain assumptions (Tessler et al., 2018).”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “Recent work in RL safety (Dalal et al. 2018; Achiam et al. 2017; Tessler et al. 2018; Satija et al. 2020) has cast safety in the context of Constrained MDPs (CMDPs) (Altman 1999), and we will concentrate on pre-defined constraints on the environment in this context.”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, neutral): “Chen et al. [20] propose the Reward Constrained Policy Optimization (RCPO) that is a multi-timescale algorithm for safe RL, and [20] show the asymptotical convergence of RCPO.”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “To achieve safety policy, a growing number of approaches have been developed over the last few decades, such as primal-dual methods [51, 152, 160, 207, 235, 239, 271, 301], trust region with safety constraints methods [4, 32, 33, 106, 283, 307], formal methods [15, 23, 25, 86, 87, 120, 189, 208, 229”
Projection-Based Constrained Policy Optimization (2020, background): “Another approach is to add a hyperparameter weighted copy of the constraints to the objective function (Tessler et al., 2018).”
Challenges of Real-World Reinforcement Learning (2019, background): “There are other approaches (Achiam et al., 2017; Tessler et al., 2018; Bohez et al., 2019) that learn a policy that violates constraints during training but produce a trained policy that respects the safety constraints.”
Let's Keep It Safe: Designing User Interfaces that Allow Everyone to Contribute to AI Safety (2019, background): “[Tessler, Mankowitz, and Mannor 2018] Tessler, C.; Mankowitz, D. J.; and Mannor, S. 2018.”
Aligning to Social Norms and Values in Interactive Narratives (2022, background): “Value alignment and safe exploration for RL agents are usually cast as constrained optimization problems wherein an agent attempts to learn a policy for a task while adhering to a given set of constraints (García et al., 2015; Leike et al., 2017; Achiam et al., 2017; Tessler et al., 2019; Ray et al.”
On Reward-Free Reinforcement Learning with Linear Function Approximation (2020, methodology): “…for 1) the batch RL setting [Bertsekas and Tsitsiklis, 1996] where data collection and planning are explicitly separated and 2) the setting where there are multiple reward function of interest, e.g., constrained RL [Achiam et al., 2017, Altman, 1999, Miryooseﬁ et al., 2019, Tessler et al., 2018].”
MuZero with Self-competition for Rate Control in VP9 Video Compression (2022, methodology): “However, as mentioned in previous works (e.g., [Tessler et al., 2018]), multi-timescale stochastic approximation is very sensitive to learning rates and is therefore typically difficult to tune.”
Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges (2021, methodology): “Typical approaches to solve the constrained MDP problem (32) include the Lagrangian methods [93], [171], constrained policy update rules [123], etc.”
Constrained Reinforcement Learning Has Zero Duality Gap (2019, methodology): “In particular, the proofs in [7, 11] rely on the fact that this different time-scale is such that allows to consider the multiplier as constant.”
An empirical investigation of the challenges of real-world reinforcement learning (2020, methodology): “There are other approaches [Achiam et al. 2017; Tessler et al. 2018; Bohez et al. 2019] that learn a policy that violates constraints during training but produce a trained policy that respects the safety constraints.”
Learning to Walk via Deep Reinforcement Learning (2018, background): “Our formulation is an instance of constrained MDP, which has been studied recently in [3, 6, 47].”
Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones (2020, methodology): “…algorithms for safe RL involves modifying the policy optimization procedure of standard RL algorithms to simultaneously reason about both task reward and constraints using methods such as trust regions [1], optimizing a Lagrangian relaxation [25, 30, 34], or constructing Lyapunov functions [6, 7].”
Reward-Free Exploration for Reinforcement Learning (2020, background): “In practice, however, reward functions are often iteratively engineered to encourage desired behavior via trial and error (e.g. in constrained RL formulations [Altman, 1999, Achiam et al., 2017, Tessler et al., 2018, Miryooseﬁ et al., 2019]).”