Learning to Score Behaviors for Guided Policy Optimization

By Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Anna Choromanska, Krzysztof Marcin Choromanski, Michael I. Jordan. 2019

46 citations recorded by the source corpus.

We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space. We show that by utilizing the dual formulation of the WD, we can learn score functions over policy behaviors that can in turn be used to lead policy optimization towards (or away from) (un)desired behaviors. Combined with smoothed WDs, the dual formulation allows us to devise efficient algorithms that take stochastic gradient descent steps through WD regularizers. We incorporate these regularizers into two novel on-policy algorithms, Behavior-Guided Policy Gradient and Behavior-Guided Evolution Strategies, which we demonstrate can outperform existing methods in a variety of challenging environments. We also provide an open source demo.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, background): “…feature map is a decomposed embedding, where φ(τ) = ∑H h=1 φ(sh, ah) and φ : S ×A → Rd is a mapping from state-actions pairs to Rd. Examples of these trajectory embeddings can be borrowed from the Behavior Guided class of algorithms for policy optimization found in Pacchiano et al. (2020).”
Effective Diversity in Population-Based Reinforcement Learning (2020, background): “This requirement can be considered as a constraint on the behavior of the new policy (Pacchiano et al., 2019).”
Primal Wasserstein Imitation Learning (2020, background): “(2017) show that replacing the f -divergence by the Wasserstein distance through the Kantorovich-Rubinstein duality (Villani, 2008) leads to better training stability, which a number of methods in IL have leveraged (Li et al., 2017; Lacotte et al., 2019; Kostrikov et al., 2019; Xiao et al., 2019; Li”