Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization

By Chelsea Finn, Sergey Levine, Pieter Abbeel. 2016

1,059 citations recorded by the source corpus.

Reinforcement learning can acquire complex behaviors from high-level specifications. However, defining a cost function that can be optimized effectively and encodes the correct task is challenging in practice. We explore how inverse optimal control (IOC) can be used to learn behaviors from demonstrations, with applications to torque control of high-dimensional robotic systems. Our method addresses two key challenges in inverse optimal control: first, the need for informative features and effective regularization to impose structure on the cost, and second, the difficulty of learning the cost function under unknown dynamics for high-dimensional continuous systems. To address the former challenge, we present an algorithm capable of learning arbitrary nonlinear cost functions, such as neural networks, without meticulous feature engineering. To address the latter challenge, we formulate an efficient sample-based approximation for MaxEnt IOC. We evaluate our method on a series of simulated tasks and real-world robotic manipulation problems, demonstrating substantial improvement over prior methods both in terms of task complexity and sample efficiency.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “This ﬁts into a recent trend of scaling reward learning methods to large deep learning systems, for example inverse RL (Finn et al., 2016), imitation learning (Ho and Ermon, 2016; Stadie et al., 2017), semi-supervised skill generalization (Finn et al., 2017), and bootstrapping RL from…”
One-Shot Imitation Learning (2017, background): “…Pomerleau (1989); Ross et al. (2011)); and inverse reinforcement learning (Ng & Russell, 2000), where a reward function (Abbeel & Ng, 2004; Ziebart et al., 2008; Levine et al., 2011; Finn et al., 2016; Ho & Ermon, 2016) is estimated that explains the demonstrations as (near) optimal behavior.”
Scalable agent alignment via reward modeling: a research direction (2018, background): “This includes trajectory preferences (Christiano et al., 2017; Kreutzer et al., 2018), goal state examples (Bahdanau et al., 2018), demonstrations (Finn et al., 2016; Ho & Ermon, 2016), as well as combinations thereof (Tung et al., 2018; Ibarz et al., 2018).”
Supervising strong learners by amplifying weak experts (2018, background): “…box (RL from a human training signal) is beginning to be explored, and includes inverse reinforcement learning (Ng and Russell, 2000; Abbeel and Ng, 2004; Finn et al., 2016) and RL from human feedback (Knox and Stone, 2009; Pilarski et al., 2011; MacGlashan et al., 2017; Christiano et al., 2017).”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “As discussed by Finn et al., this process corresponds to a generative adversarial network over trajectories, and also corresponds exactly to maximum entropy inverse reinforcement learning (Finn et al., 2016a).”
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, methodology): “Finally, the use of demonstration data has played an important role in robot learning, both as a means to obtain suitable cost functions [2, 14, 4, 8] but also to bootstrap and thus speed up learning.”
Generative Adversarial Imitation Learning (2016, methodology): “Using expressive cost function classes, like Gaussian processes [14] and neural networks [6], is crucial to properly explain complex expert behavior without meticulously hand-crafted features.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “Model-based DRL methods are also utilized by Finn et al. (2016) and Tzeng et al. (2015), learning useful state representations for generating successful control policies.”
Deep Q-learning From Demonstrations (2017, methodology): “Demonstrations have also been used for inverse optimal control in high-dimensional, continuous robotic control problems (Finn, Levine, and Abbeel 2016).”
A Framework for Data-Driven Robotics (2019, background): “Reward learning: Learning reward functions using inverse RL dates back to Ng and Russell [35] and has achieved tremendous success recently [36]–[42].”
Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation (2018, background): “iseases and complex medication space. Methods of combing SL and RL utilize expert behaviors to accelerate reinforcement learning and avoid the risks of actions. Common examples are imitation learning [1, 12, 19, 28, 44] and supervised actor-critic [5, 6, 10]. Given the samples of expert trajectories”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, background): “Inverse reinforcement learning (e.g., see [Abbeel and Ng, 2004, Ng et al., 2000, Ziebart et al., 2008, Finn et al., 2016a,b, Fu et al., 2017]) is another important and successful line of ideas for imitation learning.”
Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards (2017, methodology): “Guided Cost Learning (GCL) [1] and Generative Adversarial Imitation Learning (GAIL) [4] are the ﬁrst efﬁcient imitation learning algorithms to learn from high-dimensional inputs without knowledge of the dynamics and hand-crafted features.”
Hyperparameter Selection for Imitation Learning (2021, background): “Therefore, the learning of a reward function, which introduces its own set of extra HPs, is intertwined with the learning process of a direct RL agent (Finn et al., 2016; Ho & Ermon, 2016; Kostrikov et al., 2018).”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “Sometimes polices are learned with a frozen encoder [2], [9], [10], other times in conjunction with end-to-end training [13].”
Benchmarking Model-Based Reinforcement Learning (2019, background): “In this paper, we use the MD-GPS variant [35].”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “Data-driven approaches for reward specification [29, 1, 11, 17, 12, 48, 33, 9, 27, 4, 18] seek to overcome this issue, but typically require demonstration data to acquire rewards.”
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation (2017, background): “…within imitation learning are behavioral cloning, which performs supervised learning from observations to actions (e.g., [4], [24]) and inverse reinforcement learning [25], where a reward function [26], [27], [28], [29], [30] is estimated to explain the demonstrations as (near) optimal behavior.”
One-Shot Visual Imitation Learning via Meta-Learning (2017, background): “However, this requires additional robot experience to optimize the reward [19, 20, 21].”