Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning

By Wen Sun, J. Bagnell, Byron Boots. 2018

100 citations recorded by the source corpus.

In this paper, we propose to combine imitation and reinforcement learning via the idea of reward shaping using an oracle. We study the effectiveness of the near-optimal cost-to-go oracle on the planning horizon and demonstrate that the cost-to-go oracle shortens the learner's planning horizon as function of its accuracy: a globally optimal oracle can shorten the planning horizon to one, leading to a one-step greedy Markov Decision Process which is much easier to optimize, while an oracle that is far away from the optimality requires planning over a longer horizon to achieve near-optimal performance. Hence our new insight bridges the gap and interpolates between imitation learning and reinforcement learning. Motivated by the above mentioned insights, we propose Truncated HORizon Policy Search (THOR), a method that focuses on searching for policies that maximize the total reshaped reward over a finite planning horizon when the oracle is sub-optimal. We experimentally demonstrate that a gradient-based implementation of THOR can achieve superior performance compared to RL baselines and IL baselines even when the oracle is sub-optimal.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “require further interactions with the environment to compensate (Hester et al., 2017; Sun et al., 2018; Cheng et al., 2018).”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “For example, function approximators (critics) have been adopted to (partially) replace the Monte Carlo estimates of accumulated costs to reduce variance but at the expense of introducing bias in the search direction [2, 11–15].”
Deep Imitative Models for Flexible Inference, Planning, and Control (2018, background): “While our target setting is ofﬂine data collection, online imitation learning is an active area of research in the case of hybrid IL-RL (Ross & Bagnell, 2014; Sun et al., 2018) and “safe” IL (Sun et al., 2017; Menda et al., 2017; Zhang & Cho, 2017).”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, neutral): “We refer to [Kurutach et al., 2018, Clavera et al., 2018, Sun et al., 2018b, Chua et al., 2018, Sanchez-Gonzalez et al., 2018, Pascanu et al., 2017, Khansari-Zadeh and Billard, 2011, Luo et al., 2018] and the reference therein for recent work on model-based RL. Off-policy reinforcement learning…”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “While the combination of demonstrations and reinforcement has been studied extensively in single task problems [20, 38, 29, 22], this combination is particularly important in meta-learning contexts where few-shot learning of new tasks is simply not possible without demonstrations.”
Optimal Completion Distillation for Sequence Learning (2018, methodology): “Our work is inspired by Imitation Learning techniques (Ross et al., 2011; Ross and Bagnell, 2014; Sun et al., 2018), where a student policy is optimized to imitate an expert teacher.”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “Sun et al. [54] propose and arrive at a similar bound in the context of imitation learning and reward shaping.”
Imitation Learning as f-Divergence Minimization (2019, methodology): “Even though IL has of late been used to bootstrap reinforcement learning (RL) [9, 37, 41, 44, 45], we focus on the original problem where an extrinsic reward is not defined.”
Guided Meta-Policy Search (2019, unknown): “…efﬁcient in the single-task setting (Peters & Schaal, 2006; Kober & Peters, 2009; Kormushev et al., 2010; Taylor et al., 2011; Brys et al., 2015; Subramanian et al., 2016; Hester et al., 2018; Sun et al., 2018; Rajeswaran et al., 2018; Nair et al., 2018; Kober et al., 2013; Silver et al., 2016).”
Provable Representation Learning for Imitation Learning via Bi-level Optimization (2020, background): “More advanced approaches have also been proposed (Ross et al., 2011; Ross & Bagnell, 2014; Sun et al., 2018).”
Goal-conditioned Imitation Learning (2019, background): “Unfortunately most work in the field cannot outperform the expert, unless another reward is available during training [27, 28, 29], which might defeat the purpose of using demonstrations in the first place.”