Towards Resolving Unidentifiability in Inverse Reinforcement Learning

By Kareem Amin, Satinder Singh. 2016

34 citations recorded by the source corpus.

We consider a setting for Inverse Reinforcement Learning (IRL) where the learner is extended with the ability to actively select multiple environments, observing an agent's behavior on each environment. We first demonstrate that if the learner can experiment with any transition dynamics on some fixed set of states and actions, then there exists an algorithm that reconstructs the agent's reward function to the fullest extent theoretically possible, and that requires only a small (logarithmic) number of experiments. We contrast this result to what is known about IRL in single fixed environments, namely that the true reward function is fundamentally unidentifiable. We then extend this setting to the more realistic case where the learner may not select any transition dynamic, but rather is restricted to some fixed set of environments that it may try. We connect the problem of maximizing the information derived from experiments to submodular function maximization and demonstrate that a greedy algorithm is near optimal (up to logarithmic factors). Finally, we empirically validate our algorithm on an environment inspired by behavioral psychology.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Risks from Learned Optimization in Advanced Machine Learning Systems (2019, methodology): “Unidentifiability in mesa-optimization is partially analogous to the problem of unidentifiability in reward learning, in that the central issue is identifying the “correct” objective function given particular training data.[20] We discuss this relationship further in section 5.”
Occam's razor is insufficient to infer the preferences of irrational agents (2017, background): “Amin and Singh [2016] categorise the problem into representational and experimental unidentifiability.”
Repeated Inverse Reinforcement Learning (2017, background): “This paper extends an unpublished arXiv paper by the authors [1].”
Avoiding Wireheading with Value Reinforcement Learning (2016, background): “Inverse reinforcement learning (IRL) (also known as Bayesian inverse planning) studies how preferences and utility functions can be inferred from the actions of other agents (Ng and Russell, 2000; Sezener, 2015; Evans et al., 2016; Amin and Singh, 2016).”
Objective Robustness in Deep Reinforcement Learning (2021, background): “In their seminal paper on IRL, Ng and Russell [28] note that identifying the exact reward function an optimal agent optimizes with its behavior is in general impossible since there may be inﬁnitely many reward functions consistent with any observed policy π in a ﬁxed environment.”