Preparing for the Unknown: Learning a Universal Policy with Online System Identification

By Wenhao Yu, C. Karen Liu, Greg Turk. 2017

368 citations recorded by the source corpus.

We present a new method of learning control policies that successfully operate under unknown dynamic models. We create such policies by leveraging a large number of training examples that are generated using a physical simulator. Our system is made of two components: a Universal Policy (UP) and a function for Online System Identification (OSI). We describe our control policy as universal because it is trained over a wide array of dynamic models. These variations in the dynamic model may include differences in mass and inertia of the robots' components, variable friction coefficients, or unknown mass of an object to be manipulated. By training the Universal Policy with this variation, the control policy is prepared for a wider array of possible conditions when executed in an unknown environment. The second part of our system uses the recent state and action history of the system to predict the dynamics model parameters mu. The value of mu from the Online System Identification is then provided as input to the control policy (along with the system state). Together, UP-OSI is a robust control policy that can be used across a wide range of dynamic models, and that is also responsive to sudden changes in the environment. We have evaluated the performance of this system on a variety of tasks, including the problem of cart-pole swing-up, the double inverted pendulum, locomotion of a hopper, and block-throwing of a manipulator. UP-OSI is effective at these tasks across a wide range of dynamic models. Moreover, when tested with dynamic models outside of the training range, UP-OSI outperforms the Universal Policy alone, even when UP is given the actual value of the model dynamics. In addition to the benefits of creating more robust controllers, UP-OSI also holds out promise of narrowing the Reality Gap between simulated and real physical systems.

Subfields: Artificial Intelligence, Control and Systems Engineering, Mechanical Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Mechanical Engineering

In Reinforcement Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “Rather than relying on controller robustness, Yu et al. [53] use a model trained on varied physics to perform system identiﬁcation using online trajectory data, but their approach is not shown to succeed in the real world.”
CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, methodology): “Similar parametric formulations of different environments were used in experiments in the generalization for RL literature, which have played an important role in advancing the field (Packer et al., 2018; Rajeswaran et al., 2017; Pinto et al., 2017; Yu et al., 2017; Henderson et al., 2017a; DulacArn”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “…and policies that transfer across domains, including transfer from simulation to the real world [1, 33, 34], transfer across different dynamics [35, 36, 37, 2], transfer across robot morphologies with invariant feature spaces [38] and modularity [39], transfer across viewpoints through recurrent…”
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, background): “[23] propose to not only train a policy on a distribution of simulated parameters, but also learn a component that predicts the system parameters from the current states and actions, and use the prediction as an additional input to the policy.”
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, methodology): “[50] performed online system identification explicitly on some physical parameters, such as mass, while Peng et al.”
Learning dexterous in-hand manipulation (2018, background): “This idea has also been extended to dynamics randomization [4, 61, 68] to learn a robust policy that transfers to similar environments but with different dynamics.”
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017, methodology): “Yu et al. [24] trained a system identiﬁcation module to explicitly predict parameters of interest, such as mass and friction.”
Graph networks as learnable physics engines for inference and control (2018, methodology): “Our inference model shares similar aims as approaches for learning system identification explicitly (Yu et al., 2017; Peng et al., 2017), learning policies that are robust to hidden property variations (Rajeswaran et al.”
Asymmetric Actor Critic for Image-Based Robot Learning (2017, background): “Making policies robust for physics adaptation [26]–[28] is also receiving interest, but these methods haven’t been shown to be powerful enough to work on real robots.”
Learning Awareness Models (2018, methodology): “Yu et al. (2017) propose a method of learning control policies that operate under unknown dynamics models.”
Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, background): “There have been more robotics works that do not use vision, but that apply domain randomization on physical properties of the simulator to aid transferability [40, 46, 1, 68, 43].”
Assessing Generalization in Deep Reinforcement Learning (2018, methodology): “…problem and is under active investigation (Wang et al., 2016; Duan et al., 2016b; Rajeswaran et al., 2017a; Pinto et al., 2017; Finn et al., 2017; Kansky et al., 2017; Yu et al., 2017; Sung et al., 2017; Leike et al., 2017; Al-Shedivat et al., 2018; Clavera et al., 2018; Sæmundsson et al., 2018).”
AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning (2021, background): “Moreover, a set of approaches focus on sim2real transfer by adapting the parameters (Yu et al., 2017; Peng et al., 2020).”
RMA: Rapid Motor Adaptation for Legged Robots (2021, background): “5) System Identification [55]: Instead of predicting ẑt, we directly predict the system parameters êt.”
Causal Induction from Visual Observations for Goal Directed Tasks (2019, neutral): “This problem has been previously studied in in the context of domain adaptation (Tzeng et al., 2015; Peng et al., 2017), system identiﬁcation (Yu et al., 2017; Zhou et al., 2019), meta-learning (Finn et al., 2017; Sæmundsson et al., 2018), and multi-task learning (Andrychowicz et al., 2017; Schaul…”