Approximate MaxEnt Inverse Optimal Control and Its Application for Mental Simulation of Human Interactions

By De-An Huang, Amir-massoud Farahmand, Kris M. Kitani, J. Bagnell. 2015

29 citations recorded by the source corpus.

Maximum entropy inverse optimal control (MaxEnt IOC) is an effective means of discovering the underlying cost function of demonstrated human activity and can be used to predict human behavior over low-dimensional state spaces (i.e., forecasting of 2D trajectories). To enable inference in very large state spaces, we introduce an approximate MaxEnt IOC procedure to address the fundamental computational bottleneck stemming from calculating the partition function via dynamic programming. Approximate MaxEnt IOC is based on two components: approximate dynamic programming and Monte Carlo sampling. We analyze this approximation approach and provide a finite-sample error upper bound on its excess loss. We validate the proposed method in the context of analyzing dual-agent interactions from video, where we use approximate MaxEnt IOC to simulate mental images of a single agents body pose sequence (a high-dimensional image space). We experiment with sequences image data taken from RGB and RGBD data and show that it is possible to learn cost functions that lead to accurate predictions in high-dimensional problems that were previously intractable.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “, 2015), and forecasting of human behavior (Huang and Kitani, 2014; Huang et al., 2015).”
Forecasting Interactive Dynamics of Pedestrians with Fictitious Play (2016, background): “There has been growing interest in developing computational models of human activities that can extrapolate unseen information and predict future unobserved activities [34, 38, 22, 33, 18, 42, 40, 37, 15, 14, 16, 17, 35, 9, 49, 43, 36, 32].”
Iterative Value-Aware Model Learning (2018, methodology): “This type of error propagation analysis has been performed before by Munos [2007], Antos et al. [2008], Farahmand et al. [2010], Scherrer et al. [2012], Huang et al. [2015], Mann et al. [2015], Farahmand et al. [2016c].”
An Invitation to Imitation (2015, background): “[Huang et al., 2015 ] has developed a variant of the maximum entropy IOC that relies on a combination of function approximation of the log-partition function and sampling to estimate the gradient.”
Value-Aware Loss Function for Model-based Reinforcement Learning (2017, methodology): “Lemma 7 (Lemma 4 of Huang et al. [2015]—Extended Version) .”