Continuous Inverse Optimal Control with Locally Optimal Examples

By Sergey Levine, Vladlen Koltun. 2012

343 citations recorded by the source corpus.

Inverse optimal control, also known as inverse reinforcement learning, is the problem of recovering an unknown reward function in a Markov decision process from expert demonstrations of the optimal policy. We introduce a probabilistic inverse optimal control algorithm that scales gracefully with task dimensionality, and is suitable for large, continuous domains where even computing a full policy is impractical. By using a local approximation of the reward function, our method can also drop the assumption that the demonstrations are globally optimal, requiring only local optimality. This allows it to learn from examples that are unsuitable for prior methods.

Subfields: Artificial Intelligence, Control and Systems Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “…(Levine et al., 2011; Wulfmeier et al., 2015), and extensions to approximate inference via the Laplace approximation (under known dynamics) (Levine and Koltun, 2012; Dragan et al., 2013) and approximate reinforcement learning (under unknown dynamics) (Finn et al., 2016b; Fu et al., 2018).”
Learning Trajectory Preferences for Manipulators via Iterative Improvement (2013, background): “Some later works in LfD provided ways for handling noisy demonstrations, under the assumption that demonstrations are either near optimal [39] or locally optimal [22].”
Generative Adversarial Imitation Learning (2016, methodology): “Scaling IRL methods to large environments has thus been the focus of much recent work [6, 13].”
Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization (2016, methodology): “…relax this requirement, for example by learning a value function instead of a cost (Todorov, 2006), solving an approximate local con-trol problem (Levine & Koltun, 2012; Dragan & Srinivasa, 2012), generating a discrete graph of states (Byravan et al., 2015; Monfort et al., 2015), or only…”
An Algorithmic Perspective on Imitation Learning (2018, background): “…learning (IRL) [Russell, 1998], also called inverse optimal control [Kalman, 1964, Moylan and Anderson, 1973, Dvijotham and Todorov, 2010, Levine and Koltun, 2012], inverse planning [Baker et al., 2009], or structural estimation of MDPs Rust [1994] the learner tries to recover a reward…”
Model-Free Imitation Learning with Policy Optimization (2016, methodology): “Highway driving Finally, we ran IM-TRPO on a variation of the highway driving task of Levine & Koltun (2012).”
Spatial adaption of robot trajectories based on laplacian trajectory editing (2015, methodology): “Indirect adaption requires a cost function calculated from a set of demonstrations to be valid over the task space (Levine and Koltun 2012; Mombaur et al. 2013).”
Recent Advances in Robot Learning from Demonstration (2020, methodology): “In the continuous case, the methods become very akin to inverse optimal control (IOC) where a hidden objective function for optimal control is estimated from demonstrations (114, 115, 116, 117), and both terms are often used equivalently in the literature.”
Learning preferences for manipulation tasks from online coactive feedback (2015, background): “Noisy demonstrations and other forms of user feedback: Some later works in LfD provided ways for handling noisy demonstrations, under the assumption that demonstrations are either near optimal, as in Ziebart et al. [62], or locally optimal, as in Levine et al. [36].”
Reward-rational (implicit) choice: A unifying formalism for reward learning (2020, methodology): “Later work computes the MLE instead [19, 22, 36] and approximates the partition function (the denominator) by a quadratic approximation about the demonstration [37], a Laplace approximation [38], or importance sampling [21].”
Active Preference-Based Learning of Reward Functions (2017, background): “Inverse Reinforcement Learning (IRL) [15, 19, 23] enables us to learn R H through demonstrated trajectories.”
A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress (2018, methodology): “s this computation by requiring system dynamics to have state value function bases, which are dicult to construct as compared to commonly used reward function bases. As an alternative, Levine et al. [61] avoid repeated forward learning by using a local approximation of the reward function around exp”
Information gathering actions over human internal state (2016, unknown): “(17)”
Planning and Decision-Making for Autonomous Vehicles (2018, background): “Continuous inverse optimal control with locally optimal examples (120) may be used to handle continuous states and actions and the fact that expert demonstrations may be noisy and possibly locally optimal.”
Planning for Autonomous Cars that Leverage Effects on Human Actions (2016, methodology): “IRL.”
Egocentric Future Localization (2016, background): “Such Markovian model is an attractive choice to encode human behaviors because it reflects the way humans make a decision [20, 23, 40].”
Learning with Humans in the Loop (2020, methodology): “We then run inverse reinforcement learning [1, 38, 67, 51] on D to obtain weights ✓.”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “Scaling IRL algorithms to larger environments has posed a signiﬁcant challenge, despite their success in producing policies mirroring expert behavior [64], [71], [72].”
RAIL: Risk-Averse Imitation Learning (2017, background): “Risk sensitivity is integral to human learning Nagen-gast, Braun, and Wolpert, Niv et al. (2010, 2012), and risk-sensitive decision-making problems, in the context of MDPs, have been investigated in various ﬁelds, e.g., in ﬁnance Ruszczy´nski (2010), operations research Howard and Matheson, Borkar…”
A Function Approximation Method for Model-based High-Dimensional Inverse Reinforcement Learning (2017, background): “[15] models the behaviors based on the local optimality of a behavior, instead of the summation of rewards.”