Phase-Parametric Policies for Reinforcement Learning in Cyclic Environments

By Arjun Sharma, Kris M. Kitani. 2018

8 citations recorded by the source corpus.

In many reinforcement learning problems, parameters of the model may vary with its phase while the agent attempts to learn through its interaction with the environment. For example, an autonomous car's reward on selecting a path may depend on traffic conditions at the time of the day or the transition dynamics of a drone may depend on the current wind direction. Many such processes exhibit a cyclic phase-structure and could be represented with a control policy parameterized over a circular or cyclic phase space. Attempting to model such phase variations with a standard data-driven approach (e.g. deep networks) without explicitly modeling the phase of the model can be challenging. Ambiguities may arise as the optimal action for a given state can vary depending on the phase. To better model cyclic environments, we propose phase-parameterized policies and value function approximators that explicitly enforce a cyclic structure to the policy or value space. We apply our phase-parameterized reinforcement learning approach to both feed-forward and recurrent deep networks in the context of trajectory optimization and locomotion problems. Our experiments show that our proposed approach has superior modeling performance than traditional function approximators in cyclic environments.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jie Tan
2. Sergey Levine
3. Danijar Hafner
4. Vikas Sindhwani
5. Tingnan Zhang
6. Atil Iscen
7. Erwin Coumans
8. Steven Bohez

In Computational Theory and Mathematics

1. Jie Tan
2. Vikas Sindhwani
3. Vincent Vanhoucke
4. Danijar Hafner

In Computer Vision and Pattern Recognition

1. Jie Tan
2. Sergey Levine
3. Jitendra Malik
4. Vikas Sindhwani
5. Danijar Hafner
6. Yunfei Bai
7. Luc Van Gool
8. Dacheng Tao

Citations in the corpus

• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, background): “Sharma and Kitani [43] exploited the cyclic nature of locomotion with phaseparametric policies.”
• Policies Modulating Trajectory Generators (2018, background): “To control locomotion, Gay et al. [15] learned a neural network that modulated a CPG. Sharma and Kitani [16] designed phase-parametric policies by exploiting the cyclic nature of locomotion.”