Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills

By Víctor Campos, Alexander R. Trott, Caiming Xiong, Richard Socher, Xavier Giró-i-Nieto, Jordi Torres. 2020

186 citations recorded by the source corpus.

Acquiring abilities in the absence of a task-oriented reward function is at the frontier of reinforcement learning research. This problem has been studied through the lens of empowerment, which draws a connection between option discovery and information theory. Information-theoretic skill discovery methods have garnered much interest from the community, but little research has been conducted in understanding their limitations. Through theoretical analysis and empirical evidence, we show that existing algorithms suffer from a common limitation -- they discover options that provide a poor coverage of the state space. In light of this, we propose 'Explore, Discover and Learn' (EDL), an alternative approach to information-theoretic skill discovery. Crucially, EDL optimizes the same information-theoretic objective derived from the empowerment literature, but addresses the optimization problem using different machinery. We perform an extensive evaluation of skill discovery methods on controlled environments and show that EDL offers significant advantages, such as overcoming the coverage problem, reducing the dependence of learned skills on the initial state, and allowing the user to define a prior over which behaviors should be learned. Code is publicly available at this https URL.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Another line of work explores learning wide behavior distribution from an offline dataset by learning a task-agnostic set of skills, either with likelihood-based approaches (Ajay et al., 2020; Campos et al., 2020; Pertsch et al., 2020; Singh et al., 2021) or by maximizing mutual information (Eysenba”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “With similar motivations, (Campos et al., 2020) propose to learn a set of skills that are state-covering rather than simply diverse.”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “Existing methods such as curiosity-driven algorithms [41–44] or maximising mutual information [45–47] have shown great promise.”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “Most other works (Achiam et al., 2018; Warde-Farley et al., 2018; Hansen et al., 2019; Campos et al., 2020) are complementary methods that can help learn a set of skills.”