PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations

By Rico Jonschkowski, Roland Hafner, Jonathan Scholz, Martin A. Riedmiller. 2017

68 citations recorded by the source corpus.

We propose position-velocity encoders (PVEs) which learn---without supervision---to encode images to positions and velocities of task-relevant objects. PVEs encode a single image into a low-dimensional position state and compute the velocity state from finite differences in position. In contrast to autoencoders, position-velocity encoders are not trained by image reconstruction, but by making the position-velocity representation consistent with priors about interacting with the physical world. We applied PVEs to several simulated control tasks from pixels and achieved promising preliminary results.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised state representation learning with robotic priors: a robustness benchmark (2017, neutral): “Robotic Priors : The term of prior in Bayesian statistics refers to the prior probability distribution, but as in the articles from [2],[3], [4] and [1], we use this term as a reference to an a priori knowledge we have about the world and not to a probability distribution.”
The limits and potentials of deep learning for robotics (2018, background): “Different variants of deep learning have been shown to successfully learn predictive physics models and robot con-trol policies in a purely data driven way [2], [17], [51], [127].”
Visual Reinforcement Learning with Imagined Goals (2018, methodology): “These methods use the learned representation as a substitute for the state for the policy, but require additional information, such as access to the ground truth reward function based on the true state during training time [16, 14, 48, 11, 21, 17], expert trajectories [44], human demonstrations [42]”
State Representation Learning for Control: An Overview (2018, methodology): “The following formula from [ Jonschkowski et al., 2017] makes it explicit: where cov ( a t,i , s t +1 ,i ) is the covariance between the state s t +1 ,i at dimension i and time t and a t,i (the action at dimension i that led to such state).”
Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based robotics (2018, methodology): “• Robotic Priors The method described in (Jonschkowski et al., 2017) that encodes prior knowledge about the world as losses 3 .”
SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Control (2018, methodology): “Lastly, our losses are physically motivated, similar to those proposed for training position-velocity encoders [13], but our learned pose embedding is significantly more structured and we train our networks end-to-end directly for control.”
S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning (2018, background): “This knowledge can account for temporal continuity or causality principles that reﬂect the interactions of an agent with its environment (Jonschkowski and Brock, 2015; Jonschkowski et al., 2017).”
Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning (2022, background): “…invariance to irrelevant factors through the use of domain knowledge about the relevant state features for a task (Jonschkowski and Brock, 2015; Jonschkowski et al., 2017), via data augmentations and contrastive learning (Laskin et al., 2020a), dynamics (Watter et al., 2015; Gelada et al.,…”
SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Planning and Control (2017, unknown): “Last, our losses are physically motivated similar to those proposed for training position-velocity encoders [13], but our learned pose embedding is signiﬁcantly more structured and we train our networks end-to-end directly for control.”
Reinforcement learning for control: Performance, stability, and deep approximators (2018, background): “Another example is provided by Jonschkowski et al. (2017) who learned to encode the positions and velocities of relevant objects in an unsupervised manner.”
Learning Actionable Representations from Visual Observations (2018, methodology): “[13] introduce the reference point prior in their work in addition to the priors introduced in [4].”
Unsupervised Visuomotor Control through Distributional Planning Networks (2019, background): “Prior works have aimed to learn representations for control through auto-encoding [27, 48, 15, 16, 34], pretrained supervised features [40], spatial structure [15, 16, 23], and viewpoint invariance [41].”
A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning (2018, result): “This differs from the conclusions of [4] and [11], and we attribute our success to the presence of joint velocity in the observation, and to the last-inch task setting.”
Universal Planning Networks (2018, background): “There has been work in learning state representations usable for model-free RL when provided rewards (Lange et al., 2012; Jonschkowski & Brock, 2015; Jonschkowski et al., 2017; Higgins et al., 2017; de Bruin et al., 2018).”