Loss is its own Reward: Self-Supervision for Reinforcement Learning

By Evan Shelhamer, Parsa Mahmoudieh, Max Argus, Trevor Darrell. 2016

196 citations recorded by the source corpus.

Reinforcement learning optimizes policies for expected cumulative reward. Need the supervision be so narrow? Reward is delayed and sparse for many tasks, making it a difficult and impoverished signal for end-to-end optimization. To augment reward, we consider a range of self-supervised tasks that incorporate states, actions, and successors to provide auxiliary losses. These losses offer ubiquitous and instantaneous supervision for representation learning even in the absence of reward. While current results show that learning from reward alone is feasible, pure reinforcement learning methods are constrained by computational and data efficiency issues that can be remedied by auxiliary losses. Self-supervised pre-training and joint optimization improve the data efficiency and policy returns of end-to-end reinforcement learning.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning with Deep Energy-Based Policies (2017, background): “Pretraining has also been explored in the context of RL (Shelhamer et al., 2016).”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, methodology): “Several methods propose improving data efficiency of RL algorithms using self-supervised prediction based auxiliary tasks (Jaderberg et al., 2017; Shelhamer et al., 2017).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2016), inverse dynamics prediction (Shelhamer et al., 2016), and latent observation prediction (Guo et al.”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, methodology): “Other than learning representations through forward models, there are also works which propose to learn inverse models, in which an action is predicted based on the representations of its preceding state and subsequent state (Pathak et al., 2017; Shelhamer et al., 2016).”
Reinforcement Learning from Imperfect Demonstrations (2018, background): “(Jaderberg et al., 2016) and (Shelhamer et al., 2016) obtained better initialization via auxiliary task losses (e.g., predicting environment dynamics) in a self-supervised manner; policy performance is still initially random with these approaches.”
Continual Learning for Robotics (2019, background): “Self-supervised approaches [121, 86, 170, 149] also help to automatically explore environments.”
Multi-task Deep Reinforcement Learning with PopArt (2018, background): “Several agents (Jaderberg et al. 2016; Lample and Chaplot 2016; Shelhamer et al. 2016; Mirowski et al. 2016) demonstrated this by improving performance on a single external task by learning off-policy about auxiliary tasks deﬁned on the same stream of experience (e.g. pixel control, immediate…”
CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020, methodology): “Self-Supervised Learning for RL: Auxiliary tasks such as predicting the future conditioned on the past observation(s) and action(s) (Jaderberg et al., 2016; Shelhamer et al., 2016; van den Oord et al., 2018; Schmidhuber, 1990) are a few representative examples of using auxiliary tasks to improve the”
State Representation Learning for Control: An Overview (2018, methodology): “In these games, states can be represented through diﬀerent variables (time in achieving a task, amount of bonus, keeping alive, etc.) [Shelhamer et al., 2017, Oh et al., 2017].”
A survey and critique of multiagent deep reinforcement learning (2018, background): “, facilitating and regularizing the representation learning process [31,288]; their formalization in RL is related to the concept of general value functions [315,317].”
Discovery of Useful Questions as Auxiliary Tasks (2019, background): “Other recent examples of auxiliary tasks include depth and loop closure classiﬁcation (Mirowski et al., 2017), observation reconstruction, reward prediction, inverse dynamics prediction (Shelhamer et al., 2017), and many-goals learning (Veeriah et al., 2018).”
Reinforcement learning for control: Performance, stability, and deep approximators (2018, background): “Besides the knowledge that observations are highly redundant, other priors include the fact that the state of the world only changes slowly over time (Wiskott and Se-jnowski, 2002), as well as the fact that the state should be predictive of immediate received rewards (Shelhamer et al., 2016).”
Many-Goals Reinforcement Learning (2018, background): “The baselines for evaluating our approach of using many-goals as auxiliary task is a conventional A2C agent (without any auxiliary task), a pixel-control agent (Jaderberg et al., 2017) and a reward-prediction agent (Jaderberg et al., 2017; Shelhamer et al., 2017).”
Learning Action Representations for Reinforcement Learning (2019, methodology): “, 2018) or to obtain intrinsic rewards (Shelhamer et al., 2016; Pathak et al., 2017).”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies (2018, methodology): “We compare against several stateof-the-art feature methods, including dynamic modeling [53, 68, 36], curiosity [58], DARLA [33], and ImageNet pretraining [42], enumerated in Figure 10.”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Active Tasks (2018, methodology): “y controlling the False Discovery Rate (Q= 20%) with Benjamini-Hochberg [8]. 0 1 2 Blind Visual-Target Navigation 0:0 0:5 1:0 Visual Exploration ImageNet [42] VAE [19] DARLA [33] Forward [53] Inverse [68] Forward Inverse [2] Curiosity [58] Scratch Mid-Level 0:8 1:0 Local Planning Figure 10. Comparis”
Masked Visual Pre-training for Motor Control (2022, background): “In particular, in addition to learning the task, learn to predict some property of the environment, e.g., depth, that may lead to learning good representations as a side effect (Mirowski et al., 2017; Jaderberg et al., 2017; Shelhamer et al., 2017; Lample & Chaplot, 2017).”
A Deep Policy Inference Q-Network for Multi-Agent Systems (2017, methodology): “Auxiliary tasks are combined with DRL by learning additional goals [19, 25, 30].”