Self-supervised Learning of Image Embedding for Continuous Control

By Carlos Florensa, Jonas Degrave, Nicolas Manfred Otto Heess, Jost Tobias Springenberg, Martin A. Riedmiller. 2019

56 citations recorded by the source corpus.

Operating directly from raw high dimensional sensory inputs like images is still a challenge for robotic control. Recently, Reinforcement Learning methods have been proposed to solve specific tasks end-to-end, from pixels to torques. However, these approaches assume the access to a specified reward which may require specialized instrumentation of the environment. Furthermore, the obtained policy and representations tend to be task specific and may not transfer well. In this work we investigate completely self-supervised learning of a general image embedding and control primitives, based on finding the shortest time to reach any state. We also introduce a new structure for the state-action value function that builds a connection between model-free and model-based methods, and improves the performance of the learning algorithm. We experimentally demonstrate these findings in three simulated robotic tasks.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “Unlike prior methods that formulate an adversarial game around goal reaching [42, 36, 17], our method does not involve setting or reaching goals, but rather uses a single task to focus the forward policy, providing opportunities for the reset skills to learn varied adversarial perturbations.”
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, neutral): “We refer to [44, 45, 46, 47] for recent works towards this challenge.”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, methodology): “While most methods (Florensa et al., 2019; Savinov et al., 2018b; Wu et al., 2018) simply perform RL on top of the learned representation, our method explicitly performs search using the learned metric.”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “However, these approaches typically assume that tasks are speciﬁed to the agent at test time via mechanisms like one-hot task selectors ∗ Equal contribution [37, 43], goal images [29, 11, 26], or target conﬁgurations of the state space [14].”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, background): “This ﬁnal stage can be seen as training a policy that mimics the decoder within the MDP, i.e. a policy that will visit the state that the decoder would generate for each latent code z .”
Skew-Fit: State-Covering Self-Supervised Reinforcement Learning (2019, methodology): “Florensa et al. (2018b) samples goals from a GAN based on the difficulty of reaching the goal.”
Goal-conditioned Imitation Learning (2019, methodology): “This combination is slightly tricky because now the ﬁtted Q φ does not have the same clear interpretation it has when only one of the two rewards is used [14] .”
Grounding Language in Play (2020, methodology): “However, like other methods that combine relabeling with image observations, (Warde-Farley et al., 2018; Nair et al., 2018; Pong et al., 2019; Nair et al., 2019; Florensa et al., 2019), LfP requires that tasks be speciﬁed using a goal image to reach.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Like prior work, we will parametrize the value function as an inner product between learned representations [33, 57, 106].”