Policy Optimization via Importance Sampling

By Alberto Maria Metelli, Matteo Papini, Francesco Faccio, Marcello Restelli. 2018

103 citations recorded by the source corpus.

Policy optimization is an effective reinforcement learning approach to solve continuous control tasks. Recent achievements have shown that alternating online and offline optimization is a successful choice for efficient trajectory reuse. However, deciding when to stop optimizing and collect new trajectories is non-trivial, as it requires to account for the variance of the objective function estimate. In this paper, we propose a novel, model-free, policy search algorithm, POIS, applicable in both action-based and parameter-based settings. We first derive a high-confidence bound for importance sampling estimation; then we define a surrogate objective function, which is optimized offline whenever a new batch of trajectories is collected. Finally, the algorithm is tested on a selection of continuous control tasks, with both linear and deep policies, and compared with state-of-the-art policy optimization methods.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, background): “This is diﬀerent from value function or policy learning from oﬀ-policy data [24,25,28,32,35], where the major goal is to ensure stability and convergence.”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, background): “(Metelli et al., 2018) Let P and Q be two probability measures on the measurable space ( X, F ) such that P Q and d 2 ( P || Q ) + ∞ .”
POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning (2020, background): “First, we add an extra term in the objective encouraging larger effective sample size (ESS) and hence lower variance, following Metelli et al. (2018). Our final objective includes an ESS penalty with weight λESS > 0:”
Statistically Efficient Off-Policy Policy Gradients (2020, unknown): “Various work has investigated incorporating offline information into online policy gradients (Gu et al., 2017; Metelli et al., 2018).”
Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization (2021, methodology): “As in (Metelli et al. 2018), we use a Cantelli’s inequality on the random variable J T,α,β p ρ q , p ρ q ´ E r J T,α,β p ρ qs ě λ 1”