State Representation Learning in Robotics: Using Prior Knowledge about Physical Interaction

By Rico Jonschkowski, Oliver Brock. 2014

55 citations recorded by the source corpus.

State representations critically affect the effectiveness of learning in robots. In this paper, we propose a roboticsspecific approach to learning such state representations. Robots accomplish tasks by interacting with the physical world. Physics in turn imposes structure on both the changes in the world and on the way robots can effect these changes. Using prior knowledge about interacting with the physical world, robots can learn state representations that are consistent with physics. We identify five robotic priors and explain how they can be used for representation learning. We demonstrate the effectiveness of this approach in a simulated slot car racing task and a simulated navigation task with distracting moving objects. We show that our method extracts task-relevant state representations from highdimensional observations, even in the presence of task-irrelevant distractions. We also show that the state representations learned by our method greatly improve generalization in reinforcement learning.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

End-to-End Training of Deep Visuomotor Policies (2015, methodology): “nts, control from images has been addressed with image features (Jodogne and Piater, 2007), nonparametric methods (van Hoof et al., 2015), and unsupervised state-space learning (Bohmer et al., 2013; Jonschkowski and Brock, 2014). CNNs have also been trained to play video games with Q-learning, Monte”
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, background): “Enforcing temporal slowness during learning has previously been found to be a good proxy for learning representations in reinforcement learning [39, 40], and representation learning from videos [41].”
Unsupervised state representation learning with robotic priors: a robustness benchmark (2017, neutral): “Robotic Priors : The term of prior in Bayesian statistics refers to the prior probability distribution, but as in the articles from [2],[3], [4] and [1], we use this term as a reference to an a priori knowledge we have about the world and not to a probability distribution.”
Deep spatial autoencoders for visuomotor learning (2015, background): “The state of a robotic system should be consistent with properties of the physical world [2]; i.”
Deep predictive policy training using reinforcement learning (2017, background): “In a more recent study, Jonschkowski and Brock [8] proposed learning a task-relevant state representation based on prior physical knowledge.”
S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning (2018, methodology): “Static & random target mobile navigation: This setting simulates a navigation task using a small car resembling the task of (Jonschkowski and Brock, 2014), with either a cylinder or a horizontal band on the ground as a goal, which can be fixed or moving from episode to episode.”
Shared Autonomy with Learned Latent Actions (2020, background): “Latent actions refer to low-dimensional representations of high-dimensional actions that are learned through dimensionality reduction techniques [19, 20, 17].”
Learning Visual Feature Spaces for Robotic Manipulation with Deep Spatial Autoencoders (2015, background): “dict the dynamics of the state representation, which is not necessarily inherently possible with unconstrained learned features [2].”
Learning state representation for deep actor-critic control (2016, methodology): “In [7], these and several other assumptions about the structure of a good state representation are combined into the so-called Robotic Priors.”
Learning Semantic Embedding Spaces for Slicing Vegetables (2019, background): “Previous works in robotics have looked into learning visual representations for various tasks [5], [6], [7].”
Autonomous Learning of State Representations for Control: An Emerging Field Aims to Autonomously Learn State Representations for Reinforcement Learning Agents from Their Real-World Sensor Observations (2015, background): “Future works may train deep auto-encoder networks similar to [25], that enforce a suitable metric on the representation layer, similar to [16, 17].”
Learning state representations with robotic priors (2015, background): “Simple uniform weighting also worked well in all of our experiments [14].”
Learning latent actions to control assistive robots (2021, unknown): “Our insight here is that humans share some underlying expectations of how the control mapping should behave [27].”
Contextual Learning (2020, unknown): “…samples in representation s. Candidates for σ are the margin-based σ(d) = max(0, m − d 2 ) for some pre-defined margin m (Hadsell et al., 2006), the exponential of the negative distance σ(d) = e −d (Jonschkowski & Brock, 2014), or the Gaussian function σ(d) = e −d 2 (Jonschkowski & Brock, 2015).”
Learning User-Preferred Mappings for Intuitive Robot Control (2020, background): “Importantly, incorporating these different loss terms — which are inspired by human priors over controllable spaces [10] — enables the robot to generalize the labeled human data (which it performs supervised learning on) to unlabeled states (which it can now perform semi-supervised learning on)!”