Constrained reinforcement learning from intrinsic and extrinsic rewards

By Eiji Uchibe, Kenji Doya. 2007

61 citations recorded by the source corpus.

The main objective of a standard reinforcement learner is usually defined as maximization of a scalar reward function given externally from the environment. On the other hand, an intrinsically motivated reinforcement learner creates an intrinsic reward function from its own criteria such as curiosity, prediction error, and learning progress. This paper proposes a novel approach to deal with both intrinsic and extrinsic rewards for reinforcement learning from a viewpoint of constrained optimization problem. The extrinsic rewards construct inequality constraints to the stochastic policy while the intrinsic reward determines the current objective function for the learning agent. By integrating policy gradient reinforcement learning algorithms and techniques used in nonlinear programming, our proposed method, named the constrained policy gradient reinforcement learning (CPGRL), maximizes the long-term average intrinsic reward under the inequality constraints induced by the extrinsic rewards. The CPGRL is successfully applied to a simple MDP problem and a control task of a robot arm.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Pieter Abbeel
3. Shimon Whiteson
4. Chelsea Finn
5. J. Urgen Schmidhuber
6. Satinder Singh
7. Ruslan Salakhutdinov
8. Doina Precup

In Computational Theory and Mathematics

1. Shimon Whiteson
2. Diederik Marijn Roijers
3. Pieter Abbeel
4. Aviv Tamar
5. Satinder Singh
6. Jonathan Sorg
7. David Held
8. Andrew Barto

In Electrical and Electronic Engineering

1. Pieter Abbeel
2. Shimon Whiteson
3. Aviv Tamar
4. Jonathan Sorg
5. David Held
6. Satinder Singh
7. Andrew Barto
8. Diederik Marijn Roijers

In Reinforcement Learning

1. Sergey Levine
2. Pieter Abbeel
3. Shimon Whiteson
4. Aviv Tamar
5. Chelsea Finn
6. Satinder Singh
7. Ofir Nachum
8. Doina Precup

Citations in the corpus

• Constrained Policy Optimization (2017, methodology): “Heuristic algorithms for policy search in CMDPs have been proposed (Uchibe & Doya, 2007), and approaches based on primal-dual meth-ods can be shown to converge to constraint-satisfying policies (Chow et al., 2015), but there is currently no approach for policy search in continuous CMDPs that…”
• Intrinsically Motivated Reinforcement Learning: An Evolutionary Perspective (2010, background): “Following their earlier work [50], these authors treat extrinsic rewards as constraints on learning, while intrinsic rewards set the learning objective.”
• Intrinsically motivated information foraging (2010, background): “While an intrinsic reward can also be task specific (indeed, much of the prior work on intrinsic rewards uses them in combination with external rewards to speed up –or just get better policies from– reinforcement learning problems with external goals [17], [18]), in this paper we focus on the intrin”
• A Survey of Multi-Objective Sequential Decision-Making (2013, methodology): “Uchibe and Doya (2009) also propose a policy-gradient method for MORL called Constrained Policy Gradient RL (CPGRL) which uses a gradient projection technique to find policies whose average reward satisfies constraints on one or more of the objectives.”