Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations

By Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, Scott Niekum. 2019

425 citations recorded by the source corpus.

A critical flaw of existing inverse reinforcement learning (IRL) methods is their inability to significantly outperform the demonstrator. This is because IRL typically seeks a reward function that makes the demonstrator appear near-optimal, rather than inferring the underlying intentions of the demonstrator that may have been poorly executed in practice. In this paper, we introduce a novel reward-learning-from-observation algorithm, Trajectory-ranked Reward EXtrapolation (T-REX), that extrapolates beyond a set of (approximately) ranked demonstrations in order to infer high-quality reward functions from a set of potentially poor demonstrations. When combined with deep reinforcement learning, T-REX outperforms state-of-the-art imitation learning and IRL methods on multiple Atari and MuJoCo benchmark tasks and achieves performance that is often more than twice the performance of the best demonstration. We also demonstrate that T-REX is robust to ranking noise and can accurately extrapolate intention by simply watching a learner noisily improve at a task over time.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

A General Language Assistant as a Laboratory for Alignment (2021, background): “Learning from human preferences [CLB + 17] and T-REX IRL [BGNN19] learn from ranked data.”
Adversarial Training for High-Stakes Reliability (2022, background): “Most empirical work on techniques for aligning powerful AI [12, 13, 14, 15, 16] has focused on achieving good average-case performance in domains where no single action is catastrophic, for example using human trajectory rankings [17, 18, 19] or imitation learning [20, 21].”
Reward (Mis)design for Autonomous Driving (2021, background): “Methods of learning reward functions include inverse reinforcement learning from demonstrations [36,60], learning reward functions from preferences over trajectory segments [57,10,7], and inverse reward design from trial-and-error reward design in multiple instances of a task domain [19].”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, methodology): “…when the reward is parameterized by a neural network (Liu et al., 2009; Xia et al., 2008; Cao et al., 2007; Christiano et al., 2017a; Ouyang et al., 2022; Brown et al., 2019; Shin et al., 2023; Busa-Fekete et al., 2014; Wirth et al., 2016, 2017; Christiano et al., 2017b; Abdelkareem et al., 2022).”
The Importance of Pessimism in Fixed-Dataset Policy Optimization (2020, positive): “when learning from suboptimal demonstrations [3].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “2) Inverse Optimal Control: Traditional IOC algorithms learn an unknown reward function R ( s, a ) from expert demonstrations, where the expert’s reward function can be represented as a linear combination of features [50], [61], [62], [63], [64].”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, positive): “Brown et al. [113] proposed an approach to IRL from observation, speciﬁcally tailored to extrapolating the expert’s intentions from suboptimal ranked demonstrations.”
Scaling data-driven robotics with reward sketching and batch reinforcement learning (2019, background): “It can be done by whole episode comparisons [2, 3, 12, 59] or shorter clip comparisons [13, 29].”
A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress (2018, methodology): “[44] show that if the input to IRL also includes a ranking of the trajectories based on the degree of suboptimality of the trajectory, we may utilize this additional preference information to train a neural network representation of the reward function.”
Primal Wasserstein Imitation Learning (2020, background): “…environment’s reward) they oﬀer the beneﬁt of being less sensitive to distributional shift [Pomerleau, 1991], generalizing to environments with possibly changing dynamics [Piot et al., 2013], or recovering a near-optimal agent from suboptimal demonstrations [Brown et al., 2019, Jacq et al., 2019].”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, methodology): “In order to encourage a feature encoding that has information easily interpretable via linear combinations, the temporal difference, T-REX, inverse dynamics, and forward dynamics tasks consist of only a single layer atop the feature encoding space rather than multiple layers.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Some prior RL methods have also used contrastive learning to acquire reward functions [13, 19, 32, 37, 63, 67, 92, 132, 133, 146], often in imitation learning (3)Our focus will not be on recent methods that learn representations without negative samples [17, 42].”
Policy Gradient Bayesian Robust Optimization for Imitation Learning (2021, methodology): “Given such input, there are typically many reward functions that are consistent with it; however, many reward inference algorithms (Fu et al., 2017; Finn et al., 2016; Brown et al., 2019) will output only one of them—not necessarily the true reward.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “Additionally, the quality of demonstrations is limited by the talent of whatever expert is providing them, which warrants more research on learning from suboptimal human demonstrations (e.g., Brown et al. [2019], Zhang et al. [2021]).”