Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamics

By Michael Herman, Tobias Gindele, Jörg Wagner, Felix Schmitt, Wolfram Burgard. 2016

79 citations recorded by the source corpus.

Inverse Reinforcement Learning (IRL) describes the problem of learning an unknown reward function of a Markov Decision Process (MDP) from observed behavior of an agent. Since the agent's behavior originates in its policy and MDP policies depend on both the stochastic system dynamics as well as the reward function, the solution of the inverse problem is significantly influenced by both. Current IRL approaches assume that if the transition model is unknown, additional samples from the system's dynamics are accessible, or the observed behavior provides enough samples of the system's dynamics to solve the inverse problem accurately. These assumptions are often not satisfied. To overcome this, we present a gradient-based IRL approach that simultaneously estimates the system's dynamics. By solving the combined optimization problem, our approach takes into account the bias of the demonstrations, which stems from the generating policy. The evaluation on a synthetic MDP and a transfer learning task shows improvements regarding the sample efficiency as well as the accuracy of the estimated reward functions and transition models.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chao Yu
2. Jiming Liu
3. Shamim Nemati
4. Hongyi Zhao
5. Guoqi Ren

Citations in the corpus

• Reinforcement Learning in Healthcare: A Survey (2019, background): “e during the policy learning and reward learning can amplify the bias in each learning process, potentially leading to divergent solutions that can be of little use in practical clinical applications [313], [131]. Last but not the least, while it is possible to define a short-term reward function at ”
• Inverse reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units (2019, background): “Some theoretical research on IRL [17, 29] has investigated these issues recently and can be investigated in the clinical settings here.”