Divide-and-Conquer Reinforcement Learning

By Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, Sergey Levine. 2017

135 citations recorded by the source corpus.

Standard model-free deep reinforcement learning (RL) algorithms sample a new initial state for each trial, allowing them to optimize policies that can perform well even in highly stochastic environments. However, problems that exhibit considerable initial state variation typically produce high-variance gradient estimates for model-free RL, making direct policy or value function optimization challenging. In this paper, we develop a novel algorithm that instead partitions the initial state space into "slices", and optimizes an ensemble of policies, each on a different slice. The ensemble is gradually unified into a single policy that can succeed on the whole state space. This approach, which we term divide-and-conquer RL, is able to solve complex tasks where conventional deep RL methods are ineffective. Our results show that divide-and-conquer RL greatly outperforms conventional policy gradient methods on challenging grasping, manipulation, and locomotion tasks, and exceeds the performance of a variety of prior methods. Videos of policies learned by our algorithm can be viewed at this http URL

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

Progress & Compress: A scalable framework for continual learning (2018, methodology): “Dis-tillation is also used by Ghosh et al. (2017) to composite multiple low-level RL skills, and by Furlanello et al. (2016) to maintain performance on multiple sequential supervised tasks through a transfer learning paradigm.”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, background): “As described in Ghosh et al. [33], it is often difﬁcult to learn multiple tasks together with on-policy policy gradient methods, because of high variance and conﬂicting gradients.”
Gradient Surgery for Multi-Task Learning (2020, methodology): “…as a subroutine of the algorithm before distilling the independent models into a multi-tasking model (Levine et al., 2016; Parisotto et al., 2015; Rusu et al., 2016a; Ghosh et al., 2017; Teh et al., 2017), producing a multi-task model but losing out on the efﬁciency gains over independent training.”
Learning by Playing - Solving Sparse Reward Tasks from Scratch (2018, background): “These include for example: shaping rewards (Ng et al., 1999; Randløv & Alstrøm, 1998; Gu et al., 2017), curriculum learning (Heess et al., 2017; Ghosh et al., 2018; Forestier et al., 2017), transfer of learned policies from simulation to reality (see Duan et al. (2017); Sadeghi et al. (2017); Tobin…”
Reward-Conditioned Policies (2019, background): “…learning in the inner loop of a reinforcement learning update, either by imitating a computational expert (e.g., another more local RL algorithm) [7, 20], the best-performing trajectories [25], or by reweighting sub-optimal data to make it resemble samples from a more optimal policy [31, 32].”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, background): “Generalizing between different initial conditions is known to be difﬁcult (Ghosh et al., 2017; Langlois et al., 2019; Zolna et al., 2019).”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “However, since the targets can change rapidly, the variance in gradient estimates can be very high for function approximation methods [25].”
Hindsight policy gradients (2017, background): “Learning such goal-conditional behavior has received significant attention in machine learning and robotics, especially because a goal-conditional policy may generalize desirable behavior to goals that were never encountered by the agent (Schmidhuber & Huber, 1990; Da Silva et al., 2012; Kupcsik et ”
Algorithmic Improvements for Deep Reinforcement Learning applied to Interactive Fiction (2019, background): “The use of additional context variables has by now been demonstrated in a number of settings (Rakelly et al. (2019); Icarte et al. (2018); Ghosh et al. (2018)).”
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale (2021, methodology): “Combining multiple task policies has been explored in reinforcement learning by using gating networks [76, 50], conditioning policies on tasks [15], mapping tasks to parameters of a policy [13, 38, 82], distilling separate task policies into a shared multi-task policy [40, 77, 62, 53, 27, 5].”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, background): “To retain the role of default policy as a regularizer, it has been explored to restrict its input (Galashov et al., 2019; Grau-Moya et al., 2019), param-eteric form (Czarnecki et al., 2018) or to share it across different contexts (Teh et al., 2017; Ghosh et al., 2018).”
KeyIn: Discovering Subgoal Structure with Keyframe-based Video Prediction (2019, background): “Indeed, much of the difﬁculty in learning complex tasks lies in determining which subtask to solve, as the subpolicies that solve each subtask can often be more easily modeled individually (Teh et al., 2017; Ghosh et al., 2018).”
On the Weaknesses of Reinforcement Learning for Neural Machine Translation (2019, methodology): “Recently, diversity-based and multi-goal methods for RL were proposed to tackle similar challenges (Andrychowicz et al., 2017; Ghosh et al., 2018; Eysenbach et al., 2019).”
Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation (2020, unknown): “Distilling local policies into a global policy with a supervised objective provides a more robust way to learn from a broad distribution over domains and ultimately enable better generalisation to held-out testing domains – albeit at some additional computational cost (Ghosh et al., 2018).”
Distilling Policy Distillation (2019, background): “Many RL distillation frameworks set up knowledge transfer as a supervised learning problem [23, 24, 9, 33], by following updates in the direction of: , with Monte Carlo estimates for the expectation based on trajectories derived from the teacher policy, π .”
Composing Complex Skills by Learning Transition Policies (2018, methodology): “We follow the tasks and environment settings proposed in Ghosh et al. (2018).”
Learning to Coordinate Manipulation Skills via Skill Behavior Diversification (2020, background): “…(Schmidhuber, 1990; Bacon et al., 2017; Nachum et al., 2018; Levy et al., 2019), meta-learned (Frans et al., 2018), pre-deﬁned (Kulkarni et al., 2016; Oh et al., 2017; Merel et al., 2019; Lee et al., 2019), or attained from additional supervision signals (Andreas et al., 2017; Ghosh et al., 2018).”