Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences

By Daniel S. Brown, Russell T. Coleman, Ravi Srinivasan, Scott Niekum. 2020

116 citations recorded by the source corpus.

Bayesian reward learning from demonstrations enables rigorous safety and uncertainty analysis when performing imitation learning. However, Bayesian reward learning methods are typically computationally intractable for complex control problems. We propose Bayesian Reward Extrapolation (Bayesian REX), a highly efficient Bayesian reward learning algorithm that scales to high-dimensional imitation learning problems by pre-training a low-dimensional feature encoding via self-supervised tasks and then leveraging preferences over demonstrations to perform fast Bayesian inference. Bayesian REX can learn to play Atari games from demonstrations, without access to the game score and can generate 100,000 samples from the posterior over reward functions in only 5 minutes on a personal laptop. Bayesian REX also results in imitation learning performance that is competitive with or better than state-of-the-art methods that only learn point estimates of the reward function. Finally, Bayesian REX enables efficient high-confidence policy evaluation without having access to samples of the reward function. These high-confidence performance bounds can be used to rank the performance and risk of a variety of evaluation policies and provide a way to detect reward hacking behaviors.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Alignment & RLHF, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Alignment & RLHF

In Reinforcement Learning

Citations in the corpus

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “Another possible direction is optimizing policies against a distribution of rewards (Brown et al., 2020; Javed et al., 2021), which may prevent over-ﬁtting to a given set of metrics.”
The MAGICAL Benchmark for Robust Imitation (2020, neutral): “learning forward or inverse dynamics [9]).”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “In the inverse reinforcement learning setting, several approaches have examined how to bound the policy loss between the the robot’s policy (commonly a policy learning via RL under the inferred reward function) and the optimal policy under the ground-truth reward function of the demonstrator, even t”
Policy Gradient Bayesian Robust Optimization for Imitation Learning (2021, methodology): “However, prior work on Bayesian reward learning often only optimizes policies for the expected or MAP reward estimate over the learned posterior (Ramachandran & Amir, 2007; Choi & Kim, 2011; Brown et al., 2020a).”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “Motivated by this, many works have studied the reward specification, including inverse reinforcement learning [38] with robot demonstrations [39, 40, 41, 42, 43], learning rewards from user preferences [44, 45, 46, 47], and learning rewards from videos of humans [48, 49].”
Bayesian Robust Optimization for Imitation Learning (2020, methodology): “Future work includes taking advantage of recent research on efficient non-linear Bayesian reward learning via Gaussian processes [8] and deep neural networks [9, 11].”
Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review (2022, background): “Brown et al. (2020) propose a much faster algorithm, based on learning from preferences over demonstrations.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “Other strategies can include forms of risk-aversion [Hadfield-Menell et al., 2017] or handling uncertainty with a safe “shield” policy [Jansen et al., 2018, Srinivasan et al., 2020, Cohen and Hutter, 2020].”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “Low-dimensional state feature embedding is pretrained using ranked demonstrations [66].”