Relative Entropy Policy Search

By Jan Peters, Katharina Muelling, Yasemin Altun. 2010

829 citations recorded by the source corpus.

Policy search is a successful approach to reinforcement learning. However, policy improvements often result in the loss of information. Hence, it has been marred by premature convergence and implausible solutions. As first suggested in the context of covariant policy gradients, many of these problems may be addressed by constraining the information loss. In this paper, we continue this path of reasoning and suggest the Relative Entropy Policy Search (REPS) method. The resulting method differs significantly from previous policy gradient approaches and yields an exact update step. It can be shown to work well on typical reinforcement learning benchmark problems.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Trust Region Policy Optimization (2015, methodology): “Unlike REPS, our approach does not require a costly nonlinear optimization in the inner loop.”
Emergence of Locomotion Behaviours in Rich Environments (2017, background): “One effective measure is to employ a trust region constraint that restricts the amount by which any update is allowed to change the policy [19, 7, 14].”
Benchmarking Deep Reinforcement Learning for Continuous Control (2016, background): “Relative Entropy Policy Search (REPS) (Peters et al., 2010): This algorithm limits the loss of information per iteration and aims to ensure a smooth learning progress (Deisenroth et al., 2013).”
End-to-End Training of Deep Visuomotor Policies (2015, background): “But ﬁrst, we must choose a tractable way to represent the inﬁnite set of constraints p ( u One approach proposed in prior work on policy search for approximating such inﬁnite constraints is to replace them with expectations of features [32].”
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, neutral): “[22, 32, 44, 48]).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “Well-known examples include Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), and use conservative, KL-regularized policy updates to restrict the RL algorithm to stay close to its own prior policy (Haarnoja et al., 2018; Kakade, 2002; Peters et al., 2010; Rawlik et al., 2012).”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “This objective differs from the ones used in the derivations of related algorithms, such as RWR and REPS (Peters & Schaal, 2007; Peters et al., 2010; Abdolmaleki et al., 2018), which maximize the expected return J ( π ) instead of the expected improvement.”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “Since policies are distributions, a natural choice for the form of this constraint is a bound on the KL-divergence between the new policy and the old one (Bagnell and Schneider, 2003; Peters et al., 2010; Levine and Abbeel, 2014; Schulman et al., 2015; Abdolmaleki et al., 2018).”
Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, background): “Peters et al., 2010; Schulman et al., 2015).”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, methodology): “…previous work has used the relative entropy ( i.e., KL divergence) to regularize policy optimization; resulting in a softmax relationship between the optimal policy and state values (Peters et al., 2010; Azar et al., 2012, 2011; Fox et al., 2016; Rawlik et al., 2013) under single-step rollouts.”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, background): “Although policy updates can be improved by the use of appropriate geometry [14, 27, 32], the need for variance reduction remains paramount.”
Reinforcement Learning with Deep Energy-Based Policies (2017, methodology): “A number of related methods have also used maximum entropy policy optimization as an intermediate step for optimizing policies under a standard expected reward objective (Peters et al., 2010; Neumann, 2011; Rawlik et al., 2012; Fox et al., 2016).”
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (2016, background): “…optimization landscapes, a number of techniques have been proposed, including constraining the change in the policy at each gradient step (Kakade, 2001; Peters et al., 2010) and mixing value-based back-ups to trade off bias and variance in Monte Carlo return estimates (Schulman et al., 2015).”
PGQ: Combining policy gradient and Q-learning (2016, methodology): “An alternative is to use KL-divergence instead of entropy as a regularizer, or as a constraint on how much deviation is permitted from a prior policy (Bagnell & Schneider, 2003; Peters et al., 2010; Schulman et al., 2015; Fox et al., 2015).”
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, methodology): “Reinforcement learning (RL) methods have been applied to range of robotic control tasks, from locomotion [1], [2] to manipulation [3], [4], [5], [6] and autonomous vehicle control [7].”
Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “Other works have used related methods for reinforcement learning [31, 32, 33, 34, 35].”
Continuous Deep Q-Learning with Model-based Acceleration (2016, methodology): “In such domains, the methods of choice have been efficient model-free algorithms that use more suitable, task-specific representations (Peters et al., 2010; Deisenroth et al., 2013), as well as model-based algorithms that learn a model of the system with supervised learning and optimize a policy und”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, background): “The analytic solution to Equation 1 can be obtained by enforcing the KKT conditions [13, 14, 12].”
Toward Interpretable-AI Policies Using Evolutionary Nonlinear Decision Trees for Discrete-Action Systems (2020, methodology): “While these methods are efﬁcient at training the AI policies for a given control system task, the developed AI policies, often represented using complicated artiﬁcial neural networks, are usually complex and noninterpretable.”