Reinforcement Learning from Imperfect Demonstrations

By Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, Trevor Darrell. 2018

211 citations recorded by the source corpus.

Robust real-world learning should benefit from both demonstrations and interactions with the environment. Current approaches to learning from demonstration and reward perform supervised learning on expert demonstration data and use reinforcement learning to further improve performance based on the reward received from the environment. These tasks have divergent losses which are difficult to jointly optimize and such methods can be very sensitive to noisy demonstrations. We propose a unified reinforcement learning algorithm, Normalized Actor-Critic (NAC), that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data. NAC learns an initial policy network from demonstrations and refines the policy in the environment, surpassing the demonstrator's performance. Crucially, both learning from demonstration and interactive refinement use the same objective, unlike prior approaches that combine distinct supervised and reinforcement losses. This makes NAC robust to suboptimal demonstration data since the method is not forced to mimic all of the examples in the dataset. We show that our unified reinforcement learning algorithm can learn robustly and outperform existing baselines when evaluated on several realistic driving games.

Subfields: Artificial Intelligence, Automotive Engineering, Economics and Econometrics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Economics and Econometrics

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Works studying the transition from ofﬂine to online generally ﬁnd that likelihood-based approaches, like our sequence modeling objective, are more successful [26, 27].”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, background): “This data can potentially come from demonstrations for the desired task [16, 2], suboptimal policies [6], demonstrations for related tasks [21], or even just random exploration in the environment.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “Gao et al. (2018) introduced an imitation learning algorithm which learned from highly imperfect demonstrations, by favoring seen actions, but is limited to discrete actions.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Learning : Several recent works (Veˇcer´ık et al., 2017; Nair et al., 2017; Gao et al., 2018; Zhu et al., 2018) have introduced algorithms that unify reinforcement learning and imitation learning such that the learning agent can beneﬁt from both expert demonstrations and interactions with the…”
Self-Imitation Learning (2018, background): “Learning from imperfect demonstrations A few studies have attempted to learn from imperfect demonstrations, such as DQfD (Hester et al., 2018), Q-ﬁlter (Nair et al., 2017), and normalized actor-critic (Xu et al., 2018).”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, background): “Condition (2) is of particular interest, as it captures many common use-cases in practice, such as learning from imperfect demonstration data (e.g., of the sort that are commonly available for autonomous driving [15]), or reusing previously collected experience during off-policy RL.”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “…et al., 2013, Chemali and Lazaric, 2015, Piot et al., 2014], and a common paradigm for continuous state and action space is to initialize with RL algorithms with a good policy or Q function [Rajeswaran et al., 2017, Nair et al., 2018, Večerík et al., 2017, Hester et al., 2018, Gao et al., 2018].”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, background): “Concurrent works with this paper have also proposed to integrate demonstrations into reward functions [33], and there have been attempts to learn with imperfect demonstrations [11].”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, methodology): “In reinforcement learning, it is common to initialize a policy from suboptimal demonstrations and then improve this policy using the ground truth reward signal (Kober & Peters, 2009; Taylor et al., 2011; Hester et al., 2017; Gao et al., 2018).”
ReFT: Reasoning with Reinforced Fine-Tuning (2024, background): “This approach enables ReFT to greatly improve generalization in math problem-solving (Gao et al., 2018; Brown et al., 2020).”
PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning (2018, methodology): “2) Combining RL and IL: Second, combining RL and IL has been shown to lead to faster, more stable training as well as higher-quality solutions in robot manipulation [36], [37], [38].”
Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models (2020, background): “[23] provides a unified reinforcement learning objective that also handles imperfect demonstrations through soft optimality.”
Backplay: "Man muss immer umkehren" (2018, unknown): “Imitation learning requires access to both state and expert actions (whereas Backplay only requires states) and is designed to copy an expert, thus it cannot, without further adjustments (e.g. as proposed by Gao et al. (2018)), surpass a suboptimal expert.”
Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation (2018, methodology): “We combine RL training with Imitation Learning similarly to (Hester et al., 2017; Gao et al., 2018; Oh et al., 2018), but with special considerations for the imperfect demonstration scenario.”
Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning (2020, unknown): “Demonstration data can be used to derive a policy [15] which might be further reﬁned using typical RL training [13] or to shape the rewards biasing towards previously seen expert demonstrations [4].”
The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors (2021, methodology): “In many simple games with singular tasks, such as the Atari 2600 [2], OpenAI Gym [5], and TORCS environments2, imitation learning can drastically reduce the number of environment samples needed through pretraining and hybrid RL techniques [6, 11, 15, 27].”
Aligning to Social Norms and Values in Interactive Narratives (2022, background): “Contemporary approaches often rely on imitating expert demonstrations to learn safe trajectories (Gao et al., 2018) or by modeling rewards that best fit human values (Hadfield-Menell et al.”
BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning (2019, background): “…problem (Schaal, 1999; Argall et al., 2009; Hussein et al., 2017) and also continues to be a highly active area of research (Kim et al., 2013; Piot et al., 2014; Chemali & Lazaric, 2015; Hester et al., 2018; Ho et al., 2016; Sun et al., 2017; 2018; Cheng et al., 2018; Gao et al., 2018).”
Projection-Based Constrained Policy Optimization (2020, background): “, 2017) or (2) through expert demonstrations (Ross et al., 2011; Rajeswaran et al., 2017; Gao et al., 2018).”