Boosted and reward-regularized classification for apprenticeship learning

By Bilal Piot, Matthieu Geist, Olivier Pietquin. 2014

49 citations recorded by the source corpus.

This paper deals with the problem of learning from demonstrations, where an agent called the apprentice tries to learn a behavior from demonstrations of another agent called the expert. To address this problem, we place ourselves into the Markov Decision Process (MDP) framework, which is well suited for sequential decision making problems. A way to tackle this problem is to reduce it to classification but doing so we do not take into account the MDP structure. Other methods which take into account the MDP structure need to solve MDPs which is a difficult task and/or need a choice of features which is problem-dependent. The main contribution of the paper is to extend a large margin approach, which is a classification method, by adding a regularization term which takes into account the MDP structure. The derived algorithm, called Reward-regularized Classification for Apprenticeship Learning (RCAL), does not need to solve MDPs. But, the major advantage is that it can be boosted: this avoids the choice of features, which is a drawback of parametric approaches. A state of the art experiment (Highway) and generic experiments (structured Garnets) are conducted to show the performance of RCAL compared to algorithms from the literature.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Deep Q-learning From Demonstrations (2017, methodology): “• Supervised losses: In addition to TD losses, a large margin supervised loss is applied that pushes the value of the demonstrator’s actions above the other action values (Piot, Geist, and Pietquin 2014a).”
Learning from Demonstrations for Real World Reinforcement Learning (2017, methodology): “We add a large margin classification loss Piot et al. (2014a): JE(Q) = max a∈A [Q(s, a) + l(aE , a)]−Q(s, aE) where aE is the action the expert demonstrator took in state s and l(aE , a) is a margin function that is 0 when a = aE and positive otherwise.”
Score-based Inverse Reinforcement Learning (2016, unknown): “Here, we consider a special case of Garnets specified by three parameters: (NS , NA, NB).”
Observational Learning by Reinforcement Learning (2017, background): “In order for this problem to become tractable, most of the time we need to make some structural assumptions about this reward – like linearity in a given feature space, or smoothness [27, 28].”
SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards (2019, methodology): “(Piot et al., 2014) is to regularize Qθ with a sparsity prior on the implied rewards – in particular, a penalty on the magnitude of the rewards ∑ s∈S,a∈A |Rq(s, a)| implied by Qθ via the soft Bellman equation (Equation 3), where”
Primal Wasserstein Imitation Learning (2020, background): “…estimation Another line of research in IL consists in estimating the state-action support of the expert and deﬁne a reward that encourages the agent to stay on the support of the expert [Piot et al., 2014, Schroecker and Isbell, 2017, Wang et al., 2019, Brantley et al., 2020, Reddy et al., 2020].”