Optimizing Simulations with Noise-Tolerant Structured Exploration
By Krzysztof Marcin Choromanski, Atil Iscen, Vikas Sindhwani, Jie Tan, Erwin Coumans. 2018
12 citations recorded by the source corpus.
We propose a simple drop-in noise-tolerant replacement for the standard finite difference procedure used ubiquitously in blackbox optimization. In our approach, parameter perturbation directions are defined by a family of structured orthogonal matrices. We show that at the small cost of computing a Fast Walsh-Hadamard/Fourier Transform (FWHT/FFT), such structured finite differences consistently give higher quality approximation of gradients and Jacobians in comparison to vanilla approaches that use coordinate directions or random Gaussian perturbations. We find that trajectory optimizers like Iterative LQR and Differential Dynamic Programming require fewer iterations to solve several classic continuous control tasks when our methods are used to linearize noisy, blackbox dynamics instead of standard finite differences. By embedding structured exploration in a quasi-Newton optimizer (LBFGS), we are able to learn agile walking and turning policies for quadruped locomotion, that successfully transfer from simulation to actual hardware. We theoretically justify our methods via bounds on the quality of gradient reconstruction and provide a basis for applying them also to nonsmooth problems.
Subfields: Aerospace Engineering, Artificial Intelligence, Computational Mechanics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Deepak Pathak
- Jitendra Malik
- Erwin Coumans
- Jie Tan
- Ashish Kumar
- Vincent Vanhoucke
- Ken Caluwaerts
- Tingnan Zhang
In Artificial Intelligence
- Sergey Levine
- Jitendra Malik
- Pieter Abbeel
- Trevor Darrell
- Jie Tan
- Chelsea Finn
- Quoc V. Le
- Deepak Pathak
In Computational Mechanics
- Jitendra Malik
- Vikas Sindhwani
- Deepak Pathak
- Angjoo Kanazawa
- Andreas Geiger
- Michael J. Black
- Gerard Pons-Moll
- Shahram Izadi
Citations in the corpus
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, methodology): “One way to overcome this challenge is to use black box optimization [6].”
- RMA: Rapid Motor Adaptation for Legged Robots (2021, background): “Optimizing controllers, combined with MPC, can mitigate some of the problems [30, 8, 9], however they still require significant task-specific feature engineering [11, 15, 3].”
- Policies Modulating Trajectory Generators (2018, methodology): “Popular techniques include Black-box Optimization [1], Bayesian Optimization [2], Evolutionary Algorithms [3, 4] and Reinforcement Learning [5, 6, 7].”