Residual Policy Learning

By Tom Silver, Kelsey R. Allen, Josh Tenenbaum, Leslie Pack Kaelbling. 2018

263 citations recorded by the source corpus.

We present Residual Policy Learning (RPL): a simple method for improving nondifferentiable policies using model-free deep reinforcement learning. RPL thrives in complex robotic manipulation tasks where good but imperfect controllers are available. In these tasks, reinforcement learning from scratch remains data-inefficient or intractable, but learning a residual on top of the initial controller can yield substantial improvements. We study RPL in six challenging MuJoCo tasks involving partial observability, sensor noise, model misspecification, and controller miscalibration. For initial controllers, we consider both hand-designed policies and model-predictive controllers with known or learned transition models. By combining learning with control algorithms, RPL can perform long-horizon, sparse-reward tasks for which reinforcement learning alone fails. Moreover, we find that RPL consistently and substantially improves on the initial controllers. We argue that RPL is a promising approach for combining the complementary strengths of deep reinforcement learning and robotic control, pushing the boundaries of what either can achieve independently. Video and code at this https URL.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

TossingBot: Learning to Throw Arbitrary Objects With Residual Physics (2019, background): “concurrent work on residual reinforcement learning [16, 30] in block-assembly and object manipulation tasks.”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, background): “…(Schaal, 1996; Peters & Schaal, 2006; Ko-rmushev et al., 2010; Hester et al., 2017; Vecer´ık et al., 2017; Nair et al., 2018; Rajeswaran et al., 2018; Silver et al., 2018; Peng et al., 2018; Johannink et al., 2019; Gupta et al., 2019), this usually requires collecting demonstrations for the…”
Learning Visual Affordances for Robotic Manipulation (2019, background): “These benefits are also observed in concurrent work on residual reinforcement learning [54, 108] in block-assembly and object manipulation tasks.”
Residual Reinforcement Learning for Robot Control (2018, background): “Most closely related to our work, residual policy learning concurrently and independently explores training a residual policy in the context of simulated long-horizon, sparse-reward tasks with environment variation and sensor noise [32].”
Side-Tuning: Network Adaptation via Additive Side Networks (2019, methodology): “This has been successfully used in computer vision (ResNets [8]) and in robotics, where residual RL [10, 30] (ii) shows the connectivity structure when using side-tuning along with alpha-blending.”
Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks (2019, background): “Residual learning has also been explored in robotics as residual RL [10,33], in which we train an agent for a single task by first taking a coarse policy (e.”
Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models (2020, unknown): “Residual RL: Methods in this family [35], [36], [37] decompose the policy into a sum of two parts, one representing prior knowledge, namely trained from demonstration data, and one residual policy that is learned through RL.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “In residual RL (Silver et al. 2018; Tan et al. 2018; Johannink et al. 2019), the reinforcement learner learns a policy that is additively combined with the scripted policy, i.”
Boosted Curriculum Reinforcement Learning (2022, methodology): “Apart from RPL , we also evaluate probabilistic policy reuse ( PPR , Fern´andez et al. (2010)) as it – like RPL – introduces no crucial additional design choices in the curriculum.”
Residual Policy Learning for Shared Autonomy (2020, methodology): “Our work builds off of a recent advance in reinforcement learning for robotics: residual policy learning [26, 47].”