Off-Policy Policy Search

By Nicolas Meuleau, Leonid Peshkin, Leslie Pack Kaelbling, Kee-Eung Kim. 2007

18 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Sample Efficient Actor-Critic with Experience Replay (2016, background): “iency of actor-critics. However, controlling the variance and stability of off-policy estimators is notoriously hard. Importance sampling is one of the most popular approaches for offpolicy learning (Meuleau et al., 2000; Jie & Abbeel, 2010; Levine & Koltun, 2013). In our context, it proceed”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “Importance sampling is a simple strategy for incorporating off-policy data (Sutton & Barto, 1998; Meuleau et al., 2000; Hachiya et al., 2009), but can introduce optimization instabilities due to the potentially large variance of the importance sampling estimator.”
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, methodology): “…learning, since the current policy π is updated with the samples generated from old behaviour policies µ , an importance sampling (IS) ratio is used to rescale each sampled reward to correct for the sampling bias at time-step t : ρ t = π ( a t | b t ) /µ ( a t | b t ) (Meuleau et al., 2000).”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “Importance sampling (Meuleau et al., 2000; Jie & Abbeel, 2010; Levine & Koltun, 2013) is a simple way to correct for the discrepancy between π and β , but, it requires computing i π i ( a i | τ i ) β i ( a i | τ i ) , whose variance grows exponentially with the number of agents in multi-agent…”
Reinforcement learning with misspecified model classes (2013, background): “Existing off-policy PS methods [23] use importance sampling to reuse the data while attempting to avoid learning bias.”
Motor Learning at Intermediate Reynolds Number: Experiments with Policy Gradient on the Flapping Flight of a Rigid Wing (2010, background): “As the same systems on which one wishes to use these algorithms tend to have a high cost of policy evaluation, much work has been done on maximizing the policy improvement from any individual evaluation (Meuleau et al., 2000; Williams et al., 2006).”