Learning by Playing - Solving Sparse Reward Tasks from Scratch

By Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Manfred Otto Heess, Jost Tobias Springenberg. 2018

513 citations recorded by the source corpus.

We propose Scheduled Auxiliary Control (SAC-X), a new learning paradigm in the context of Reinforcement Learning (RL). SAC-X enables learning of complex behaviors - from scratch - in the presence of multiple sparse reward signals. To this end, the agent is equipped with a set of general auxiliary tasks, that it attempts to learn simultaneously via off-policy RL. The key idea behind our method is that active (learned) scheduling and execution of auxiliary policies allows the agent to efficiently explore its environment - enabling it to excel at sparse reward RL. Our experiments in several challenging robotic manipulation settings demonstrate the power of our approach.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “SAC-X [Riedmiller et al., 2018] is one example that takes advantage of multi-task learning to improve efﬁciency, by simultaneously learning the policy and a set of auxiliary tasks to explore its observation space - in search for sparse rewards of the externally deﬁned target task.”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “ its great challenges is long-term credit assignment, which is required for delayed rewards [88, 68, 38, 98]. Delayed rewards are associated with episodic and sparse rewards, and therefore are common [92]. Classical reinforcement learning methods like temporal difference (TD), Monte Carlo (MC), and ”
Structured agents for physical construction (2019, methodology): “RS0 learning algorithm : For continuous action outputs, we use an actor-critic learning algorithm that combines re-trace with stochastic value gradients (denoted RS0) (Munos et al., 2016; Heess et al., 2015; Riedmiller et al., 2018).”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “• Auxiliary Tasks (Mirowski et al., 2016; Jaderberg et al., 2016; Levine et al., 2016; Yu et al., 2018; Riedmiller et al., 2018): Uses additional supervised or unsupervised tasks (e.g., regressing depth images from color images, detecting loop closures, predicting end-effector poses) alongside the…”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, methodology): “As before, data from all executed tasks is collected in one big data-set annotating each trajectory snippet with all rewards (i.e. this is similar to the SAC-R setting from (Riedmiller et al., 2018).”
Gradient Surgery for Multi-Task Learning (2020, methodology): “The multi-task formulation has been applied to many different settings, including supervised learning [63, 35, 60, 53, 62] and reinforcement-learning [17, 58], as well as many different domains, such as vision [3, 39, 31, 33, 62], language [11, 15, 38, 44] and robotics [45, 59, 25].”
Off-Policy Evaluation via Off-Policy Classification (2019, background): “These can be interpreted as tasks where the agent can either “succeed” or “fail” in each trial, and although they form a subset of all possible MDPs, this subset is quite representative of many real-world tasks, and is actively used e.g. in robotic manipulation [15, 31].”
Learning Plannable Representations with Causal InfoGAN (2018, background): “Designing a reward function that depends on high-dimensional observations can be challenging, however, and most recent studies either relied on domains where the reward can be instrumented [33, 28, 40], or required successful demonstrations as guidance [16, 45].”
Emergent Coordination Through Competition (2019, methodology): “, 2012) which embeds soccer in a wider universe of possible environments with consistent simulated physics, already used extensively in the machine learning research community (Heess et al., 2016; 2017; Bansal et al., 2017; Brockman et al., 2016; Tassa et al., 2018; Riedmiller et al., 2018).”
Regularized Hierarchical Policies for Compositional Transfer in Robotics (2019, unknown): “Across all experiments, we utilize the data-sharing scheme of Scheduled Auxiliary Control (SAC-X) [Riedmiller et al., 2018] (see Section 3.”
Automatic Curriculum Learning For Deep RL: A Short Survey (2020, methodology): “In that case, ACL can be used to pose auxiliary tasks to the agent, gradually guiding its learning trajectory from simple to difficult tasks until the target tasks are solved [Matiisen et al., 2017; Florensa et al., 2017; Riedmiller et al., 2018; Ivanovic et al., 2018; Salimans and Chen, 2018].”
Multi-Task Reinforcement Learning with Soft Modularization (2020, background): “Researchers have shown learning with multiple objectives can make different tasks benefit from each other in robotics and RL [41, 24, 25, 28, 11, 34].”
Mastering Diverse Domains through World Models (2023, background): “= vψ(sT ) (7) Critic Learning A simple choice for the critic loss function would be to regress the λ-returns via squared error or symlog predictions.”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, methodology): “…networks as function approximators, have solved an impressive range of tasks, from complex control tasks (Lil-licrap et al., 2016; Heess et al., 2017; Riedmiller et al., 2018; Levine et al., 2016; OpenAI et al., 2018) to computer games (Mnih et al., 2015; OpenAI, 2018) and Go (Silver et al., 2016).”
Benchmarking Reinforcement Learning Algorithms on Real-World Robots (2018, background): “…are supported with one or more of (a) sufficient state information or auxiliary task-specific steps and knowledge (e.g. Levine et al. 2016, Riedmiller et al. 2018), (b) preparation in simulation (e.g. Rusu et al. 2017) (c) collaborative learning (e.g. Yahya et al. 2017), and (d) learning…”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “Incorporating demonstration data and other sources of supervision can further accelerate some of these methods (Ve ˇ cer ´ ık et al. 2017; Riedmiller et al. 2018).”
Influence-Based Multi-Agent Exploration (2019, background): “These works result in signiﬁcant progress in many challenging tasks such as Montezuma Revenge (Burda et al., 2018b), robotic manipulation (Pathak et al., 2018; Riedmiller et al., 2018), and Super Mario games (Burda et al., 2018a; Pathak et al., 2017).”
Goal-conditioned Imitation Learning (2019, background): “Nevertheless, these successes are shyly echoed in real world robotics [6, 7].”
Learning Curriculum Policies for Reinforcement Learning (2018, methodology): “Automatically designing a curriculum is an open problem that has only recently begun to be examined [5, 8, 9, 16, 19, 23].”