Skew-Fit: State-Covering Self-Supervised Reinforcement Learning

By Vitchyr H. Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, Sergey Levine. 2019

313 citations recorded by the source corpus.

Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manually-designed reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing goal reaching performance together with the entropy of the goal distribution, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. We prove that, under regularity conditions, Skew-Fit converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that combining Skew-Fit for learning goal distributions with existing goal-reaching methods outperforms a variety of prior methods on open-sourced visual goal-reaching tasks. Moreover, we demonstrate that Skew-Fit enables a real-world robot to learn to open a door, entirely from scratch, from pixels, and without any manually-designed reward function.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “One objective for achieving this proposed in prior work [20, 14, 40, 34] is the mutual information (MI) between skills and the states they visit: I(s; z) = H(s) − H(s|z).”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “Pong et al. (2020) propose a method to increase the importance of rarely sampled observation as goals.”
Vygotskian Autotelic Artificial Intelligence: Language and Culture Internalization for Human-Like AI (2022, methodology): “12 While previous methods were restricted to generate goals within the distribution of past experience (e.g. with generative models of states 10,62,127 ), imagine invents out-of-distribution goals by combining descriptions of past goals.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “The problems of automatically sampling goals and exploration [24, 35, 85, 98, 144] are orthogonal to this work.”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “Exploration methods are usually guided by intrinsic rewards that are selfgenerated by the robotic agent, and are not specific to any task [9, 47, 49, 60, 69, 81, 85, 88, 93, 110].”
Grounding Language in Play (2020, methodology): “However, like other methods that combine relabeling with image observations, (Warde-Farley et al., 2018; Nair et al., 2018; Pong et al., 2019; Nair et al., 2019; Florensa et al., 2019), LfP requires that tasks be speciﬁed using a goal image to reach.”
If MaxEnt RL is the Answer, What is the Question? (2019, background): “[38] Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine.”
Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning (2017, background): “…deep reinforcement learning agents also require millions of training steps, e.g. 2M steps in the Atari game Montezuma’s Revenge (Kulkarni et al., 2016), or 50M in a Doom-like ﬁrst-person shooter (Dosovitskiy and Koltun, 2017), including for RL-based IMGEPs Colas et al. (2019); Pong et al. (2020).”
Action and Perception as Divergence Minimization (2020, methodology): “…is often estimated by learning a density model of individual inputs as in pseudo-counts (Bellemare et al., 2016), latent variable models as in SkewFit (Pong et al., 2019), unnormalized models as in RND (Burda et al., 2018), and non-parameteric models as in reachability (Savinov et al., 2018).”
Deep Hierarchical Planning from Pixels (2022, background): “…et al., 2021), or training the low-level as a goal-conditioned policy (Andrychowicz et al., 2017; Levy et al., 2017; Nachum et al., 2018a;b; Co-Reyes et al., 2018; Warde-Farley et al., 2018; Nair et al., 2018; Pong et al., 2019; Hartikainen et al., 2019; Gehring et al., 2021; Shah et al., 2021).”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “This demonstrates a drawback of methods that rely simply on the MSE loss [7, 11, 31, 20, 41, 48, 21], as it is equivalent to assuming a particular, constant variance of the Gaussian decoding distribution.”
Learning Domain Invariant Representations in Goal-conditioned Block MDPs (2021, methodology): “In Section 4, we will train goal-conditioned RL agents with perfect alignment encoder using the Skew-Fit algorithm [13].”
Planning with Goal-Conditioned Policies (2019, background): “Our work largely neglects the question of exploration for goal-conditioned policies, and though this question has been studied in some recent works [17, 45, 59, 49], examining how exploration interacts with planning is an exciting future direction.”
On Reward-Free Reinforcement Learning with Linear Function Approximation (2020, unknown): “…Bellemare et al., 2016, Houthooft et al., 2016, Tang et al., 2017, Florensa et al., 2017, Pathak et al., 2017, Tang et al., 2017, Achiam et al., 2017, Hazan et al., 2018, Burda et al., 2018, Colas et al., 2018, Co-Reyes et al., 2018, Nair et al., 2018, Eysenbach et al., 2018, Pong et al., 2019].”
Planning to Explore via Self-Supervised World Models (2020, background): “Model-based control Model-free agents are often data-inefﬁcient (Kaelbling et al., 1996) and hard to adapt to different tasks, although one promising avenue for adapting these agents is goal-conditioned reinforcement learning (Kaelbling, 1993; Pathak et al., 2018; Pong et al., 2019).”