Dynamics-Aware Unsupervised Discovery of Skills

By Archit Sharma, Shixiang Shane Gu, Sergey Levine, Vikash Kumar, Karol Hausman. 2019

496 citations recorded by the source corpus.

Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment. A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks. However, learning an accurate model for complex dynamical systems is difficult, and even then, the model might not generalize well outside the distribution of states on which it was trained. In this work, we combine model-based learning with model-free learning of primitives that make model-based planning easy. To that end, we aim to answer the question: how can we discover skills whose outcomes are easy to predict? We propose an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics. Our method can leverage continuous skill spaces, theoretically, allowing us to learn infinitely many behaviors even for high-dimensional state-spaces. We demonstrate that zero-shot planning in the learned latent space significantly outperforms standard MBRL and model-free goal-conditioned RL, can handle sparse-reward tasks, and substantially improves over prior hierarchical RL methods for unsupervised skill discovery.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Another line of work explores learning wide behavior distribution from an offline dataset by learning a task-agnostic set of skills, either with likelihood-based approaches [34, 35, 36, 37] or by maximizing mutual information [38, 39, 40].”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “One objective for achieving this proposed in prior work [20, 14, 40, 34] is the mutual information (MI) between skills and the states they visit: I(s; z) = H(s) − H(s|z).”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “There are a number of recent works (Eysenbach et al., 2018; Nachum et al., 2018a; Sharma et al., 2019) which use unsupervised objectives to discover skills and use the discovered skills for planning (Sharma et al., 2019), few-shot imitation learning, or online RL (Eysenbach et al., 2018; Nachum et…”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, background): “This data can be collected using previously trained agents across a diverse set of tasks [33, 34], through agents autonomously exploring their environment [11, 12], via human teleoperation [35, 27, 36, 10] or any combination of these.”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “By contrast, state-space exploration methods (Pathak et al., 2017; Burda et al., 2019; Eysenbach et al., 2019; Sharma et al., 2020) only learn low-level primitives without strategy-level diversity (Baker et al., 2020).”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “…et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering reusable skills (Daniel et al., 2012; Gregor et al., 2016; Warde-Farley et al., 2018; Shankar & Gupta, 2020; Thomas et al., 2018; Sharma et al., 2020).”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, methodology): “…trained for a wide range of tasks by using manual reward speciﬁcation (Huber & Grupen, 1998; Stulp & Schaal, 2011), using relabeling techniques (Andrychowicz et al., 2017), or via unsupervised skill discovery (Daniel et al., 2016; Fox et al., 2017; Warde-Farley et al., 2018; Sharma et al., 2020).”
Skill-based Meta-Reinforcement Learning (2022, background): “through autonomous exploration (Hausman et al., 2018; Sharma et al., 2020; Dasari et al., 2019), human teleoperation (Schaal et al.”
Generalized Hindsight for Reinforcement Learning (2020, background): “Unsupervised learning of general low-level controllers has been a focus of recent research [19, 16, 62].”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, background): “Such unpredictability might not be desirable when training a metra-controller to solve a downstream task by combining the learned skills (Sharma et al., 2019).”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, methodology): “Unsupervised pre-training has been studied for extracting strong behavioral priors that can be utilized to solve downstream tasks efﬁciently in the context of RL (Daniel et al., 2016; Florensa et al., 2018; Achiam et al., 2018; Eysenbach et al., 2019; Sharma et al., 2020).”
Action and Perception as Divergence Minimization (2020, background): “DADS (Sharma et al., 2019) estimates the mutual information in input space by combining a forward predictor of skills with a contrastive bound.”
Deep Hierarchical Planning from Pixels (2022, background): “We also show that Director learns successfully across a wide range of different domains without giving task reward to the worker, underlining the generality of the approach.”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, methodology): “Skills discovered by information-theoretic meth-ods such as the ones considered in this work have also been used as primitives for Hierarchical RL (Florensa et al., 2017; Eysenbach et al., 2019; Sharma et al., 2019).”
Unsupervised Meta-Learning for Reinforcement Learning (2018, background): “…f : D φ → π information objectives can be formulated, including mutual information between single states and z (Eysenbach et al., 2018), pairs of start and end states and z (Gregor et al., 2016), and entire trajectories and z (Achiam et al., 2018; Sharma et al., 2019; Warde-Farley et al., 2018).”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “Such approaches include jointly learning the primitive skills with the high-level policy through goal generation or an “options” framework [6, 8, 59], learning primitives through intrinsic motivation or other auxiliary objectives [60, 61, 62], from demonstration behavior [63, 64, 65, 66, 67], or fro”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, methodology): “One such intrinsic reward, proposed in DADS (Sharma et al., 2019), can be derived from a variational approximation to the mutual information between the skills and next states I ( s (cid:48) ; z | s ) as: where z i ∼ p ( z ) .”