Learning an Embedding Space for Transferable Robot Skills

By Karol Hausman, Jost Tobias Springenberg, Ziyun Wang, Nicolas Manfred Otto Heess, Martin A. Riedmiller. 2018

350 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “, 2017) and soft actor-critic algorithms (Haarnoja et al., 2018b; Hausman et al., 2018).”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “rocesses with dynamic programming [9, 27, 91, 85]. Lastly, initialization with goal-conditioned policies to transfer knowledge is inspired by transfer learning [23, 86] and skill generalization in RL [7, 40, 32]. Concurrently, [26] also proposes to plan a sequence of subgoals leading to a ﬁnal desti”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “(Gregor et al., 2016; Hausman et al., 2018; Eysenbach et al., 2018) leverage principles of empowerment and discover goals based on information theoretic objectives, learning to represent the intrinsic control space of an agent in order to facilitate learning about more than a single task.”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “…Stolle & Precup, 2002; Peng et al., 2019) learn a discrete set of lower-level policies, each behaving as a primitive skill, other works (Vezhn-evets et al., 2017; Nachum et al., 2018b; 2019a; Hausman et al., 2018) learn a continuous space of primitive skills representing the lower-level policy.”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, methodology): “[12] Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller.”
RLBench: The Robot Learning Benchmark & Learning Environment (2019, background): “…of large-scale data [1] and deep learning algorithms in the ﬁeld of computer vision [2], there is now a large body of work looking at increasing the capabilities of robotic agents through the use of reinforcement learning [3], [4], meta-learning [5], [6], [7], multi-task learning [8], [9], etc.”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “Hausman et al. (2018), Florensa et al. (2017), and Gregor et al. (2016) showed that a discriminability objective is equivalent to maximizing the mutual information between the latent skill z and some aspect of the corresponding trajectory.”
Gradient Surgery for Multi-Task Learning (2020, background): “…such as vision (Bilen & Vedaldi, 2016; Misra et al., 2016a; Kokkinos, 2017; Liu et al., 2018; Zamir et al., 2018), language (Col-lobert & Weston, 2008; Dong et al., 2015; McCann et al., 2018; Radford et al., 2019) and robotics (Riedmiller et al., 2018; Wulfmeier et al., 2019; Hausman et al., 2018).”
Learning Latent Plans from Play (2019, methodology): “[18], who present a method for reinforcement learning of closely related manipulation skills, parameterized via an explicit skill embedding space.”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, background): “In classical RL, posterior sampling (Strens, 2000; Osband et al., 2013) maintains a posterior over possible MDPs and enables temporally extended exploration by acting optimally according to a sampled MDP.”
Dynamics-Aware Unsupervised Discovery of Skills (2019, methodology): “This ranges from the classic option-critic architecture (Sutton et al., 1999; Stolle & Precup, 2002; Perkins et al., 1999) to some of the more recent work (Bacon et al., 2017; Vezhnevets et al., 2017; Nachum et al., 2018; Hausman et al., 2018).”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, methodology): “Meta-MDPs were introduced by Hay et al. [2014] and applied to routing by Chang et al.”
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (2018, background): “l.,2017) but operating at a trajectory level. Several recent and concurrent works have proposed methods which are related to ours but have clear distinctions. Florensa et al.(2017);Heess et al.(2016);Hausman et al. (2018) learn stochastic neural networks to modulate low Figure 3. From left to right ”
Meta-Reinforcement Learning of Structured Exploration Strategies (2018, methodology): “Latent space models for RL have been explored in several prior works, though not in the context of meta-learning or learning exploration strategies (Hausman et al., 2018; Flo-Meta-Reinforcement rensa et al., 2017; Kolter & Ng, 2007).”
Task-Embedded Control Networks for Few-Shot Imitation Learning (2018, background): “[25] proposed learning a skill embedding space via reinforcement learning that led to speed-ups during training time.”
Emergent Coordination Through Competition (2019, methodology): “Our critic is learned using experience replay, minimizing a k-step TD-error with off-policy retrace corrections (Munos et al., 2016), using a separate target network for bootstrapping, as is also described in Hausman et al. (2018); Riedmiller et al. (2018).”
Latent Space Policies for Hierarchical Reinforcement Learning (2018, background): “While several prior works have sought to combine maximum entropy policies with learning of latent spaces (Haarnoja et al., 2017; Hausman et al., 2018) and even with learning hierarchies in small state spaces (Saxe”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, methodology): “…[52, 58, 118, 105, 25], external memory [53, 193], good initializations [50, 52], normalization schemes [74, 98], hierarchical methods [34, 11, 60, 188], capsules [144, 64], unsupervised learning techniques [52, 135, 89, 51], value or advantage functions [179], intrinsic motivation [174, 128,…”
Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning (2018, background): “A line of work [9, 5, 4, 7, 3] takes this approach in the context of intrinsic motivation.”