Covariant policy search

By Weiming Liu, Sanjiang Li, Jochen Renz. 2003

201 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Trust Region Policy Optimization (2015, neutral): “…problem to generate a policy update: maximize θ Lθold(θ) (12) subject to D ρθold KL (θold, θ) ≤ δ. Similar policy updates have been proposed in prior work (Bagnell & Schneider, 2003; Peters & Schaal, 2008b; Peters et al., 2010), and we compare our approach to prior methods in Section 7 and in…”
Benchmarking Deep Reinforcement Learning for Continuous Control (2016, background): “At each iteration, N trajectories { τ i } N i =1 are generated, where τ i = { ( s it , a it , r it ) } Tt =0 contains data collected along the i th trajectory.”
End-to-End Training of Deep Visuomotor Policies (2015, neutral): “To avoid this, we can bound the change from ˆ p ( τ ) to p ( τ ) in terms of their KL-divergence by a step size (cid:15) , resulting in the following constrained problem: This type of policy update has previously been proposed by several authors in the context of policy search [1, 19, 31, 32].”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “Since policies are distributions, a natural choice for the form of this constraint is a bound on the KL-divergence between the new policy and the old one (Bagnell and Schneider, 2003; Peters et al., 2010; Levine and Abbeel, 2014; Schulman et al., 2015; Abdolmaleki et al., 2018).”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, background): “Previous work (Kakade, 2002; Bagnell & Schneider, 2003; Peters & Schaal, 2008; Schulman et al., 2015) has applied natural gradient to policy optimization, locally improving expected reward subject to variants of d θ T F ( θ )d θ ≤ (cid:15) .”
PGQ: Combining policy gradient and Q-learning (2016, methodology): “An alternative is to use KL-divergence instead of entropy as a regularizer, or as a constraint on how much deviation is permitted from a prior policy (Bagnell & Schneider, 2003; Peters et al., 2010; Schulman et al., 2015; Fox et al., 2015).”
Reinforcement learning in robotics: A survey (2013, methodology): “Multiple policy-search methods used in robotics are based on this intuition (Bagnell and Schneider, 2003; Peters and Schaal, 2008b; Peters et al., 2010a; Kober and Peters, 2010).”
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, methodology): “…that the algorithm ﬁts well with policy gradient methods which constrain the KL divergence per policy update, such as covariant policy gradient (Bagnell & Schneider, 2003), natural policy gradient (Kakade & Langford, 2002), REPS (Peters et al., 2010), and trust-region policy optimization…”
Dual Policy Iteration (2018, methodology): “One approximation we make here is to replace the policy divergence measure D TV ( π n , π ) (note total variation distance is symmetric) with the KL-divergence D KL ( π n , π ) , which allows us to leverage Natural Gradient [13, 6].”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, methodology): “A number of pre-conditioned policy gradient methods have been developed in literature [19], [4], [35], [34], [43], [40], [44] and in principle any of them could be used.”
Deep spatial autoencoders for visuomotor learning (2015, methodology): “Using KL-divergence constraints for controller optimization was proposed in a number of prior works [33], [34], [35].”
Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization (2016, positive): “…are Gaussian, and each iteration of the policy optimization procedure satisﬁes a KL-divergence constraint of the form D KL ( q ( τ ) (cid:107) ˆ q ( τ )) ≤ (cid:15) , which prevents the policy from changing too rapidly (Bagnell & Schneider, 2003; Peters et al., 2010; Rawlik & Vijayakumar, 2013).”
Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (2017, background): “Bagnell and Schneider [3] further proved that the metric deﬁned in [11] is a covariant metric induced by the path-distribution manifold.”
Learning contact-rich manipulation skills with guided policy search (2015, background): “This type of policy update has previously been proposed by several authors in the context of policy search [26], [8], [27].”
A Brief Survey of Deep Reinforcement Learning (2017, methodology): “The idea of constraining each policy gradient update, as measured by the Kullback-Leibler (KL) divergence between the current and proposed policy, has a long history in RL [57, 4, 59, 103].”
Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction (2017, methodology): “We derive a natural gradient update procedure for imitation learning inspired by the success of natural gradient descent in RL (Kakade, 2002; Bagnell & Schneider, 2003; Schul-man et al., 2015).”
Dynamic policy programming (2010, methodology): “Also, the natural policy gradient method can be seen as a relative entropy method, in which the second-order Taylor expansion of the relative-entropy between the distribution of the states is considered as the metric for policy improvement (Bagnell and Schneider, 2003).”
Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning (2018, methodology): “10 using gradient-based update procedures (e.g., Stochastic Gradient Descent, Natural Gradient (Kakade, 2002; Bagnell & Schneider, 2003)) in the policy’s parameter space.”