Unsupervised Control Through Non-Parametric Discriminative Rewards

By David Warde-Farley, Tom Van de Wiele, Tejas D. Kulkarni, Catalin Ionescu, Steven Stenberg Hansen, Volodymyr Mnih. 2018

193 citations recorded by the source corpus.

Learning to control an environment without hand-crafted rewards or expert data remains challenging and is at the frontier of reinforcement learning research. We present an unsupervised learning algorithm to train agents to achieve perceptually-specified goals using only a stream of observations and actions. Our agent simultaneously learns a goal-conditioned policy and a goal achievement reward function that measures how similar a state is to the goal state. This dual optimization leads to a co-operative game, giving rise to a learned reward function that reflects similarity in controllable aspects of the environment instead of distance in the space of observations. We demonstrate the efficacy of our agent to learn, in an unsupervised manner, to reach a diverse set of goals on three domains -- Atari, the DeepMind Control Suite and DeepMind Lab.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “…et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering reusable skills (Daniel et al., 2012; Gregor et al., 2016; Warde-Farley et al., 2018; Shankar & Gupta, 2020; Thomas et al., 2018; Sharma et al., 2020).”
Learning Latent Plans from Play (2019, positive): “To address this, we are motivated by the idea of an agent capable of task-agnostic control : the ability to reach any reachable goal state from any current state [5].”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “…that utilized purely model-free unsupervised RL methods (Daniel et al., 2012; Florensa et al., 2017; Eysenbach et al., 2018; Gregor et al., 2016; Warde-Farley et al., 2018; Thomas et al., 2018), aims to learn diverse skills via a discriminability objective: a good set of skills is one where it…”
Unsupervised Meta-Learning for Reinforcement Learning (2018, background): “…f : D φ → π information objectives can be formulated, including mutual information between single states and z (Eysenbach et al., 2018), pairs of start and end states and z (Gregor et al., 2016), and entire trajectories and z (Achiam et al., 2018; Sharma et al., 2019; Warde-Farley et al., 2018).”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, methodology): “, 2017), or via unsupervised skill discovery (Daniel et al., 2016; Fox et al., 2017; Warde-Farley et al., 2018; Sharma et al., 2020).”
Skew-Fit: State-Covering Self-Supervised Reinforcement Learning (2019, methodology): “Other methods use data collected from a randomly initialized policy or heuristics based on data collected online to design a non-parametric (Colas et al., 2018b; Warde-Farley et al., 2018; Florensa et al., 2018a) or parametric (Péré et al.”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, background): “[62] also considers accumulation of tasks, but with a focus on goal-reaching and by maintaining a goal reservoir via heuristics that promote diversity.”
Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning (2022, unknown): “Non-parametric methods for exploration and planning with images have been considered (Warde-Farley et al., 2019; Nasiriany et al., 2019; Wang et al., 2019; Liu et al., 2020; Shah et al., 2021) and could also beneﬁt from improved representation learning.”
Deep Hierarchical Planning from Pixels (2022, background): “, 2021), or training the low-level as a goal-conditioned policy (Andrychowicz et al., 2017; Levy et al., 2017; Nachum et al., 2018a;b; Co-Reyes et al., 2018; Warde-Farley et al., 2018; Nair et al., 2018; Pong et al., 2019; Hartikainen et al., 2019; Gehring et al., 2021; Shah et al., 2021).”
Planning from Pixels using Inverse Dynamics Models (2020, methodology): “6 COMPARISON TO PRIOR METHODS Prior methods for goal-conditioned RL include DISCERN (Warde-Farley et al., 2019) and GoalConditioned Supervised Learning (GCSL) (Ghosh et al.”
Reinforcement Learning without Ground-Truth State (2019, methodology): “In other work on teaching a robot to open a door, an IMU sensor is placed on the door handle to determine the rotation angle of the handle and whether or not the door has been opened [3, 4].”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “Most other works (Achiam et al., 2018; Warde-Farley et al., 2018; Hansen et al., 2019; Campos et al., 2020) are complementary methods that can help learn a set of skills.”
Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards (2019, background): “…rewards (Schulman et al., 2017; Liu et al., 2019), methods to learn latent representations that facilitate shaped reward (Ghosh et al., 2018; Nair et al., 2018; Warde-Farley et al., 2019), and learning objectives that encourage diverse behaviors (Haarnoja et al., 2017; Eysenbach et al., 2019).”
Unsupervised Visuomotor Control through Distributional Planning Networks (2019, methodology): “Previous approaches to building control-centric goal representations include using inverse models [2, 35] and mutual information estimation [47].”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, background): “See Table 3 as a non-parametric approach to sampling from the desired p ( s ) (Warde-Farley et al., 2019).”
Grounding Language in Play (2020, methodology): “However, like other methods that combine relabeling with image observations, (Warde-Farley et al., 2018; Nair et al., 2018; Pong et al., 2019; Nair et al., 2019; Florensa et al., 2019), LfP requires that tasks be specified using a goal image to reach.”