Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective
By Anirudh Vemula, Wen Sun, J. Bagnell. 2019
46 citations recorded by the source corpus.
Black-box optimizers that explore in parameter space have often been shown to outperform more sophisticated action space exploration methods developed specifically for the reinforcement learning problem. We examine these black-box methods closely to identify situations in which they are worse than action space exploration methods and those in which they are superior. Through simple theoretical analyses, we prove that complexity of exploration in parameter space depends on the dimensionality of parameter space, while complexity of exploration in action space depends on both the dimensionality of action space and horizon length. This is also demonstrated empirically by comparing simple exploration methods on several model problems, including Contextual Bandit, Linear Regression and Reinforcement Learning in continuous control.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ching-An Cheng
- Xinyan Yan
- Byron Boots
- Bhavya Kailkhura
- Sijia Liu
- A. Hero III
- Pin-Yu Chen
- Pramod K. Varshney
Citations in the corpus
- Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “The high variance of G is due to the exploration difficulty in RL: in the worst-case, the variance of G can grow exponentially in the problem’s horizon [29, 30], as it becomes harder for the policy to visit meaningful states and get useful update information.”
- A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning: Principals, Recent Advances, and Applications (2020, methodology): “And the work [83] theoretically showed that the complexity of exploration in the action space (using policy gradients) depends on both the dimensionality of the action space and the horizon length, as opposed to, the complexity of exploration in the parameter space (using ZO methods) depends only on”