Bayesian Multitask Inverse Reinforcement Learning

By Christos Dimitrakakis, Constantin A. Rothkopf. 2011

114 citations recorded by the source corpus.

We generalise the problem of inverse reinforcement learning to multiple tasks, from multiple demonstrations. Each one may represent one expert trying to solve a different task, or as different experts trying to solve the same task. Our main contribution is to formalise the problem as statistical preference elicitation, via a number of structured priors, whose form captures our biases about the relatedness of different tasks or expert policies. In doing so, we introduce a prior on policy optimality, which is more natural to specify. We show that our framework allows us not only to learn to efficiently from multiple experts but to also effectively differentiate between the goals of each. Possible applications include analysing the intrinsic motivations of subjects in behavioural experiments and learning from multiple teachers.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

The MAGICAL Benchmark for Robust Imitation (2020, neutral): “However, it’s worth noting that unlike, say, HAMDPs [16] or CIRL games [21], IL cannot request clarification from a demonstrator if the supplied demonstrations are ambiguous, which limits its ability to learn the right objective in general.”
Imitation Learning as f-Divergence Minimization (2019, background): “[45] Christos Dimitrakakis and Constantin A Rothkopf.”
Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets (2017, methodology): “Examples from this ﬁeld include [9] and [4].”
Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review (2022, methodology): “An example of multi-task IRL is (Dimitrakakis and Rothkopf, 2011), which extends Bayesian IRL (Ramachandran and Amir, 2007) to the multi-task setting by using multiple structured priors which capture the relatedness of diﬀerent tasks on reward functions and policies.”
Nonparametric Bayesian Inverse Reinforcement Learning for Multiple Reward Functions (2012, background): “Dimitrakakis and Rothkopf [9] proposed a multi-task learning approach, generalizing the Bayesian approach to IRL [4].”
A Function Approximation Method for Model-based High-Dimensional Inverse Reinforcement Learning (2017, methodology): “For motions involving multiple tasks and varying reward functions, methods are developed in [19] and [20], which try to learn multiple reward functions.”
A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress (2018, background): “Likewise, to make a high-dimensional navigation task computationally tractable, Rothkopf et al. [64] utilize the independence between the components of a task to introduce decomposition in birl.”
Convex Optimization: Algorithms and Complexity (2015, unknown): “Dimitrakakis and Rothkopf [Dimitrakakis and Rothkopf, 2011] generalize BIRL to multi-task learning.”
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning (2018, background): “Some of these approaches include those that aim to incorporate a shared prior over reward functions through extending the Bayesian IRL (Ramachandran & Amir, 2007) framework to the multi-task setting (Dimitrakakis & Rothkopf, 2012; Choi & Kim, 2012).”
A survey of inverse reinforcement learning techniques (2012, background): “Furthermore, Dimitrakakis and Rothkopf (2011) extended the Bayesian IRL to a multitask setting.”
Inverse Reinforcement Learning via Function Approximation for Clinical Motion Analysis (2017, methodology): “functions, methods are developed in [17] and [18], which try to learn multiple reward functions.”