Fast Policy Learning through Imitation and Reinforcement

By Ching-An Cheng, Xinyan Yan, Nolan Wagener, Byron Boots. 2018

90 citations recorded by the source corpus.

Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of RL and IL. We accomplish this by formulating several popular RL and IL algorithms in a common mirror descent framework, showing that these algorithms can be viewed as a variation on a single approach. We then propose LOKI, a strategy for policy learning that first performs a small but random number of IL iterations before switching to a policy gradient RL method. We show that if the switching time is properly randomized, LOKI can learn to outperform a suboptimal expert and converge faster than running policy gradient from scratch. Finally, we evaluate the performance of LOKI experimentally in several simulated environments.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “require further interactions with the environment to compensate (Hester et al., 2017; Sun et al., 2018; Cheng et al., 2018).”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “For example, they have straightforward update rules and convergence guarantee, as they minimizes J directly by updating parameters in descent directions in expectation [2, 7–9].”
Agile Autonomous Driving using End-to-End Deep Imitation Learning (2017, neutral): “This surrogate problem is comparatively more structured than the original RL problem (1) [4], so we can adopt algorithms with provable performance guarantees.”
Imitation Learning as f-Divergence Minimization (2019, methodology): “Even though IL has of late been used to bootstrap reinforcement learning (RL) [9, 37, 41, 44, 45], we focus on the original problem where an extrinsic reward is not defined.”
BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning (2019, background): “…problem (Schaal, 1999; Argall et al., 2009; Hussein et al., 2017) and also continues to be a highly active area of research (Kim et al., 2013; Piot et al., 2014; Chemali & Lazaric, 2015; Hester et al., 2018; Ho et al., 2016; Sun et al., 2017; 2018; Cheng et al., 2018; Gao et al., 2018).”
EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints (2020, unknown): “…suggests that cost functions close to the cross-entropy loss are better suited to deep neural models than the squared error (Leblond et al., 2018; Cheng et al., 2018), we deﬁne the cost function as the KL divergence be-tween the action distributions given by the model policy and by the oracle…”
Non-Monotonic Sequential Text Generation (2019, background): “However, recent work has found that, especially when dealing with recurrent neural network policies (which we will use; see § 4), using a cost function more analogous to a cross-entropy loss can be preferred (Leblond et al., 2018; Cheng et al., 2018; Welleck et al., 2018).”
Predictor-Corrector Policy Optimization (2018, methodology): “Most existing first-order policy optimization algorithms belong to this family [28], so we can use PicCoLO to wrap these model-free algorithms into new hybrid algorithms that can robustly use (imperfect) predictive models, such as off-policy gradients and simulated gradients, to improve policy learn”