Positive-Unlabeled Reward Learning

By Danfei Xu, Misha Denil. 2019

46 citations recorded by the source corpus.

Learning reward functions from data is a promising path towards achieving scalable Reinforcement Learning (RL) for robotics. However, a major challenge in training agents from learned reward models is that the agent can learn to exploit errors in the reward model to achieve high reward behaviors that do not correspond to the intended task. These reward delusions can lead to unintended and even dangerous behaviors. On the other hand, adversarial imitation learning frameworks tend to suffer the opposite problem, where the discriminator learns to trivially distinguish agent and expert behavior, resulting in reward models that produce low reward signal regardless of the input state. In this paper, we connect these two classes of reward learning methods to positive-unlabeled (PU) learning, and we show that by applying a large-scale PU learning algorithm to the reward learning problem, we can address both the reward under- and over-estimation problems simultaneously. Our approach drastically improves both GAIL and supervised reward learning, without any additional assumptions.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “By contrast, IRL generally requires fewer expert samples, but it relies on a prohibitive amount of agent interaction with the environment [16, 39, 42] due to the need to do reinforcement learning with a learned reward function.”
What Matters for Adversarial Imitation Learning? (2021, methodology): “A myriad of improvements over the original AIL algorithm were proposed over the years [17, 31, 41, 44, 46], from changing the discriminator’s loss function [17] to switching from on-policy to off-policy agents [31].”
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills (2021, methodology): “In this case, we must learn entirely from “positive” examples (Xu & Denil, 2019; Zolna et al., 2020), which we show experimentally is challenging for current RL methods.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Some prior RL methods have also used contrastive learning to acquire reward functions [13, 19, 32, 37, 63, 67, 92, 132, 133, 146], often in imitation learning (3)Our focus will not be on recent methods that learn representations without negative samples [17, 42].”