Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Sparse Reward Environments

By Vinicius G. Goecks, Gregory M. Gremillion, Vernon J. Lawhern, J. Valasek, Nicholas R. Waytowich. 2019

34 citations recorded by the source corpus.

This paper investigates how to efficiently transition and update policies, trained initially with demonstrations, using off-policy actor-critic reinforcement learning. It is well-known that techniques based on Learning from Demonstrations, for example behavior cloning, can lead to proficient policies given limited data. However, it is currently unclear how to efficiently update that policy using reinforcement learning as these approaches are inherently optimizing different objective functions. Previous works have used loss functions, which combine behavior cloning losses with reinforcement learning losses to enable this update. However, the components of these loss functions are often set anecdotally, and their individual contributions are not well understood. In this work, we propose the Cycle-of-Learning (CoL) framework that uses an actor-critic architecture with a loss function that combines behavior cloning and 1-step Q-learning losses with an off-policy pre-training step from human demonstrations. This enables transition from behavior cloning to reinforcement learning without performance degradation and improves reinforcement learning in terms of overall performance and training time. Additionally, we carefully study the composition of these combined losses and their impact on overall policy learning. We show that our approach outperforms state-of-the-art techniques for combining behavior cloning and reinforcement learning for both dense and sparse reward scenarios. Our results also suggest that directly including the behavior cloning loss on demonstration data helps to ensure stable learning and ground future policy updates.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “…[Atkeson and Schaal, 1997]), BC has been used as a regularization for policy optimization with DDPG [Lillicrap et al., 2015, Nair et al., 2018, Goecks et al., 2020] and the natural policy gradient [Kakade, 2001, Rajeswaran et al., 2017], but with additional sophistication through modified…”
Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning (2020, methodology): “This approach has been empirically shown to have varying degrees of success [8], [15], [18], [19].”