Grounded Action Transformation for Robot Learning in Simulation

By Josiah P. Hanna, Peter Stone. 2017

124 citations recorded by the source corpus.

Robot learning in simulation is a promising alternative to the prohibitive sample cost of learning in the physical world. Unfortunately, policies learned in simulation often perform worse than hand-coded policies when applied on the physical robot. This paper proposes a new algorithm for learning in simulation — Grounded Action Transformation — and applies it to learning of humanoid bipedal locomotion. Our approach results in a 43.27% improvement in forward walk velocity compared to a state-of-the art hand-coded walk.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, methodology): “However, [32] requires a human in the loop to select the best simulation parameters, and [33] needs to fit additional models for the real robot forward dynamics and simulator inverse dynamics.”
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, background): “Hanna and Stone [16] adapted the policy using a learned action space transformation from simulation to the real world.”
Transferring Autonomous Driving Knowledge on Simulated and Real Intersections (2017, background): “Recent work has investigated grounding imperfect simulators to a robot’s behavior (Hanna & Stone, 2017) and there is evidence that transferring from simulation to real robots can be addressed by targeting the systems ability to generalize (Tobin et al., 2017).”
RMA: Rapid Motor Adaptation for Legged Robots (2021, unknown): “making the simulation more accurate [22, 49, 19].”
Legged Locomotion in Challenging Terrains using Egocentric Vision (2022, background): “This has motivated work in using RL for learning to walk in simulation [14, 32, 33, 34], and then successfully deploy them in a diverse set of real world scenarios [35, 36, 37, 38, 39, 40, 35, 41].”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, background): “…R θ : (1) Learn an inverse dynamics model that uses embeddings φ ( s t ) and φ ( s t +1 ) to predict the corresponding action a t (Torabi et al., 2018; Hanna & Stone, 2017), (2) Learn a forward dynamics model that predicts s t +1 from φ ( s t ) and a t (Oh et al., 2015; Thananjeyan et al.,…”
Learning Agile Robotic Locomotion Skills by Imitating Animals (2020, background): “Sim-to-real transfer can be facilitated by constructing more accurate simulations [58, 62], or adapting the simulator with real-world data [57, 23, 26, 36, 5].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Grounded action transformations address the physics domain gap by iteratively modifying agent-simulator interactions to better match real-world data, even when the simulator is a fixed black box [140].”
PAC Reinforcement Learning With an Imperfect Model (2018, background): “A promising approach to addressing this issue is to train an RL agent in a simulator and transfer the solution to the real environment, which is particularly relevant but not limited to robotics domains (Koos, Mouret, and Doncieux 2010; Cutler and How 2015; Hanna and Stone 2017).”