Boosted Bellman Residual Minimization Handling Expert Demonstrations

By Bilal Piot, Matthieu Geist, Olivier Pietquin. 2014

87 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Observe and Look Further: Achieving Consistent Performance on Atari (2018, methodology): “More recent approaches have explicitly combined expert demonstrations with RL data during the learning of the policy or action-value function [3, 11, 15].”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “Imitation has been combined with reinforcement learning, via learning from demonstrations methods (Kim et al., 2013; Piot et al., 2014; Chemali & Lazaric, 2015), with deep reinforcement learning extensions (Hester et al.”
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, background): “Since here we do not have a speciﬁc target value for A π ( s, a ) , we resort to the idea of large margin classiﬁcation loss [31], which forces Q π ( s, a ) to be at least a margin l m lower than the Q-value of the second best action, i.e., max a (cid:48) (cid:54) = a Q π ( s, a (cid:48) ) .”
Deep Q-learning From Demonstrations (2017, methodology): “• Supervised losses: In addition to TD losses, a large margin supervised loss is applied that pushes the value of the demonstrator’s actions above the other action values (Piot, Geist, and Pietquin 2014a).”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “Demonstrations have been widely used to improve the efficiency of RL [Kim et al., 2013, Chemali and Lazaric, 2015, Piot et al., 2014], and a common paradigm for continuous state and action space is to initialize with RL algorithms with a good policy or Q function [Rajeswaran et al., 2017, Nair et…”
Hyperparameter Selection for Imitation Learning (2021, background): “Although expert trajectories can also be useful in this case, this is not the setting of IL, but of RL with demonstrations where both the expert demonstrations and reward signals are used (Kim et al., 2013; Piot et al., 2014; Hester et al., 2018).”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “Promising recent attempts at overcoming exploration in such tasks leverage demonstrations of successful trajectories (Kim et al., 2013; Hester et al., 2017; Vecerik et al., 2017; Nair et al., 2018; Gulcehre et al., 2020), dubbed the RL with Expert Demonstrations (RLED; Piot et al., 2014).”
Reinforcement Learning from Imperfect Demonstrations (2018, methodology): “We follow the Reinforcement Learning with Expert Demonstrations (RLED) framework (Chemali & Lazaric, 2015; Kim et al., 2013; Piot et al., 2014), where both rewards and actions are available in the demonstrations.”
Learning from Demonstrations for Real World Reinforcement Learning (2017, methodology): “We add a large margin classification loss Piot et al. (2014a): JE(Q) = max a∈A [Q(s, a) + l(aE , a)]−Q(s, aE) where aE is the action the expert demonstrator took in state s and l(aE , a) is a margin function that is 0 when a = aE and positive otherwise.”
Policy Optimization with Demonstrations (2018, methodology): “Kim et al. (2013); Piot et al. (2014) are based on policy iteration and use demonstration data to shape the value function.”
Boosted Fitted Q-Iteration (2017, background): “Bellman Residual Minimization (BRM) has been extensively studied in RL literature for policy evaluation (Antos et al., 2008; Maillard et al., 2010), learning from demonstrations (Piot et al., 2014) and feature construction (e.g., Parr et al., 2007; Fard et al., 2013).”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, methodology): “Another example is the Approximate Policy Iteration with Demonstration (APID) algorithm, which was proposed by [59] and extended by [95].”
BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning (2019, background): “…problem (Schaal, 1999; Argall et al., 2009; Hussein et al., 2017) and also continues to be a highly active area of research (Kim et al., 2013; Piot et al., 2014; Chemali & Lazaric, 2015; Hester et al., 2018; Ho et al., 2016; Sun et al., 2017; 2018; Cheng et al., 2018; Gao et al., 2018).”
Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations (2018, methodology): “Appren-ticeship learning inspired many similar algorithms ( [ Syed and Schapire, 2008 ] , [ Syed et al. , 2008 ] , [ Piot et al. , 2014 ] , [ Ho et al. , 2016 ] ), Ho et al. [ Ho and Ermon, 2016a ] proposed a imitation learning method that merges inverse reinforcement learning and reinforcement…”
Direct Policy Iteration with Demonstrations (2015, background): “…from either RL or expert samples [Ng and Russell, 2000; Abbeel and Ng, 2004; Bagnell and Ross, 2010], there is a recent surge of interest in combining expert demonstrations in RL algorithms [Kim et al., 2013; Piot et al., 2014] or cost functions in imitation learning [Ross and Bagnell, 2014].”