Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
By Yuping Luo, Huazhe Xu, Tengyu Ma. 2019
14 citations recorded by the source corpus.
Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservatively-extrapolated value functions, which provably lead to policies with self-correction. We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation. We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks. We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm, which is shown to outperform significantly prior works in sample efficiency.
Subfields: Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Reinforcement Learning
- Sergey Levine
- Aviral Kumar
- George Tucker
- Shixiang Shane Gu
- Ofir Nachum
- Chelsea Finn
- Pieter Abbeel
- Aurick Zhou
Citations in the corpus
- Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “[38] learn a conservatively-extrapolated value function by enforcing a linear extrapolation property over the state-space, and a learned dynamics model to obtain policies for goal-reaching tasks.”