Convergence of Value Aggregation for Imitation Learning

By Ching-An Cheng, Byron Boots. 2018

29 citations recorded by the source corpus.

Value aggregation is a general framework for solving imitation learning problems. Based on the idea of data aggregation, it generates a policy sequence by iteratively interleaving policy optimization and evaluation in an online learning setting. While the existence of a good policy in the policy sequence can be guaranteed non-asymptotically, little is known about the convergence of the sequence or the performance of the last policy. In this paper, we debunk the common belief that value aggregation always produces a convergent policy sequence with improving performance. Moreover, we identify a critical stability condition for convergence and provide a tight non-asymptotic bound on the performance of the last policy. These new theoretical insights let us stabilize problems with regularization, which removes the inconvenient process of identifying the best policy in the policy sequence in stochastic problems.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Agile Autonomous Driving using End-to-End Deep Imitation Learning (2017, background): “The discrepancy can be explained with a recent theoretical analysis by Cheng and Boots [3], which provides a necessary and sufﬁcient condition for the convergence of the policy sequence.”
Optimal Completion Distillation for Sequence Learning (2018, background): “Cheng and Boots (2018) showed that mixing in ground truth samples is an essential regularizer for value aggregation convergence in imitation learning.”
Fast Policy Learning through Imitation and Reinforcement (2018, background): “However, despite signiﬁcant recent breakthroughs in our understanding of imitation learning (Ross et al., 2011; Cheng and Boots, 2018), the performance of IL is still highly dependent on the quality of the expert policy.”
Predictor-Corrector Policy Optimization (2018, background): “Realizing this one step improvement in AggreVaTe, however, requires solving multiple rounds of online learning, because Π does not contain all functions and the dynamics are unknown [22].”