Active Imitation Learning via Reduction to I.I.D. Active Learning

By Kshitij Judah, Alan Fern, Thomas G. Dietterich. 2012

63 citations recorded by the source corpus.

In standard passive imitation learning, the goal is to learn a target policy by passively observing full execution trajectories of it. Unfortunately, generating such trajectories can require substantial expert effort and be impractical in some cases. In this paper, we consider active imitation learning with the goal of reducing this effort by querying the expert about the desired action at individual states, which are selected based on answers to past queries and the learner's interactions with an environment simulator. We introduce a new approach based on reducing active imitation learning to i.i.d. active learning, which can leverage progress in the i.i.d. setting. Our first contribution, is to analyze reductions for both non-stationary and stationary policies, showing that the label complexity (number of queries) of active imitation learning can be substantially less than passive learning. Our second contribution, is to introduce a practical algorithm inspired by the reductions, which is shown to be highly effective in four test domains compared to a number of alternatives.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

Active Reinforcement Learning with Monte-Carlo Tree Search (2018, background): “There is also work applying active learning to tabular RL with human teachers but where human input is quite different than in the ARL model (Subramanian et al., 2016; Judah et al., 2012).”
Boosted and reward-regularized classification for apprenticeship learning (2014, methodology): “This approach has been theoretically studied in the finite horizon case [23, 18, 8] and in the infinite horizon case [14].”
Learning from Interventions: Human-robot interaction as both explicit and implicit feedback (2020, background): “This decision can be based on a threshold of actionclassifier confidence [15, 16, 17], distributional distance or discrepancy [18, 19], query by committee [20], or a combination of state novelty and historical error [21].”
SHIV: Reducing supervisor burden in DAgger using support vectors for efficient learning from demonstrations in high dimensional state spaces (2016, methodology): “Although such methods have been previously proposed for online LfD (see [8], [11] for the former and [15], [16] for the latter),”
Automated Planning and Acting (2016, background): “Research in these issues is active (for example, [608, 303]), but so far has been applied mostly in academic benchmarks such as those of the Reinforcement Learning Competition (for example, balance a vertical pole or maintain equilibrium on a bicycle).”
Imitation Learning (2017, methodology): “…to perform and evaluate imitation learning, such as video games [Thurau et al. 2004a; Gorman 2009; Mnih et al. 2015; Geisler 2002; Ross and Bagnell 2010], AI simulators [Raza et al. 2012; Aler et al. 2005], and reinforcement learning benchmarks [Judah et al. 2012; Randlov and Alstrom 1998].”