SWIRL: A sequential windowed inverse reinforcement learning algorithm for robot tasks with delayed rewards

By Sanjay Krishnan, Animesh Garg, Richard Liaw, Brijen Thananjeyan, Lauren Miller, Florian T. Pokorny, Ken Goldberg. 2018

77 citations recorded by the source corpus.

We present sequential windowed inverse reinforcement learning (SWIRL), a policy search algorithm that is a hybrid of exploration and demonstration paradigms for robot learning. We apply unsupervised learning to a small number of initial expert demonstrations to structure future autonomous exploration. SWIRL approximates a long time horizon task as a sequence of local reward functions and subtask transition conditions. Over this approximation, SWIRL applies Q-learning to compute a policy that maximizes rewards. Experiments suggest that SWIRL requires significantly fewer rollouts than pure reinforcement learning and fewer expert demonstrations than behavioral cloning to learn a policy. We evaluate SWIRL in two simulated control tasks, parallel parking and a two-link pendulum. On the parallel parking task, SWIRL achieves the maximum reward on the task with 85% fewer rollouts than Q-learning, and one-eight of demonstrations needed by behavioral cloning. We also consider physical experiments on surgical tensioning and cutting deformable sheets using a da Vinci surgical robot. On the deformable tensioning task, SWIRL achieves a 36% relative improvement in reward compared with a baseline of behavioral cloning with segmentation.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, background): “This question is one focus area of hierarchical imitation learning (HIL), where solutions [8, 9] typically try to achieve two goals: i) learn a temporal task abstraction, and ii) discover a meaningful segmentation of the demonstrations into subtasks.”
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, background): “Imitation learning methods are either ofﬂine, such as Behavioral Cloning [30, 34, 35], or online, such as Inverse Reinforcement Learning (IRL) [1, 20].”
ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation (2018, methodology): “Prior works show that data-driven IL methods improve both the sample efﬁciency of policy learning [16, 17, 18] and the performance of the trained policy [19, 20, 21].”
Recent Advances in Robot Learning from Demonstration (2020, background): “A number of recent approaches attempt to learn from demonstrations at different levels of abstraction (130, 132, 136, 149, 151, 152, 153, 154, 155).”
Multi-Level Discovery of Deep Options (2017, background): “The ﬁeld of hierarchical reinforcement learning has a long history [9, 1, 10], and has also been applied in robotics [11, 12, 13] and in the analysis of biological systems [14, 15, 16, 17, 18].”
DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations (2017, methodology): “…builds on DDO and on our prior work in task segmentation (Transition State Clustering [6] and Sequential Windowed Inverse Reinforcement Learning [8]) with two new algorithmic extensions: (1) a hybrid categorical– continuous distribution model to parametrize high-level policies that can invoke…”
Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning (2017, methodology): “Algorithms to automatically generate tensioning policies for a given cutting trajectory can assist both human surgeons [24] and automated surgical cutting procedures to improve the reliability and accuracy of surgical cutting [17, 21].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Alternative approaches to policy similarity fit models to the underlying value function [203] or the unobserved reward function that the skill behavior is implicitly maximizing [309, 214, 261, 64, 18].”