On a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient

By Jie Tang, Pieter Abbeel. 2010

90 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Sample Efficient Actor-Critic with Experience Replay (2016, background): “With richer realistic environments, the capabilities of our agents have increased and improved.”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Speciﬁcally, ﬁtting the baseline to the current batch of data and then using the updated baseline to form the estimator results in a biased gradient (Jie & Abbeel, 2010).”
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, methodology): “Asides from these more recent works, the use of off-policy samples with policy gradients has been a popular direction of research (Peshkin & Shelton, 2002; Jie & Abbeel, 2010; Degris et al., 2012; Levine & Koltun, 2013).”
From Importance Sampling to Doubly Robust Policy Gradient (2019, neutral): “To the best of our knowledge, Tang & Abbeel (2010) was the first to explicitly mention the connection between (pertrajectory) IS and PG, which corresponds to the first row of our Table 1.”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “Especially, the state dependent CVs (also known as baselines or reward reshaping [17, 19]) have been thoroughly investigated [18].”
P3O: Policy-on Policy-off Policy Optimization (2019, neutral): “Effective sample size is popularly used to measure the quality of importance sampling and to restrict the search space for parameter updates (Jie and Abbeel, 2010; Peshkin and Shelton, 2002).”
Multiagent Soft Q-Learning (2018, background): “Any infinite horizon MDP with discounted rewards can be -approximated by a finite horizon MDP using a horizon H = logγ( (1−γ)) maxs,a |R(s,a)| (Jie and Abbeel 2010) R : S × A → R is the reward function for each agent i, and T (s, a, s′) = P (s′|s, a) is the transition function, where a = 〈a1 · · · a”
Hindsight policy gradients (2017, neutral): “Importance sampling has previously been applied to policy gradient methods in order to efﬁciently reuse information obtained by earlier policies [41].”
Policy Tree: Adaptive Representation for Policy Gradient (2015, background): “The use of importance sampling in policy gradient has been studied previously (Jie and Abbeel 2010).”
Efficient iterative policy optimization (2016, background): “The update direction can be improved by using natural gradient [Amari, 1998, Kakade, 2001] and doing a line search helps in choosing a correct stepsize [Jie and Abbeel, 2010].”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “Importance sampling (Meuleau et al., 2000; Jie & Abbeel, 2010; Levine & Koltun, 2013) is a simple way to correct for the discrepancy between π and β , but, it requires computing i π i ( a i | τ i ) β i ( a i | τ i ) , whose variance grows exponentially with the number of agents in multi-agent…”
Statistically Efficient Off-Policy Policy Gradients (2020, background): “s (2015) for H= 0 and Chen et al. (2019); Fujimoto et al. (2019) for H1. Variance reduction in PG. Variance reduction has been a central topic for PG (Greensmith et al., 2004; Schulman et al., 2016; Tang and Abbeel, 2010; Wu et al., 2018). These papers generally consider a given class of estimators ”
Hierarchical Approaches for Reinforcement Learning in Parameterized Action Space (2018, background): “∗Any infinite horizon MDP with discounted rewards can be -approximated by a finite horizon MDP using a horizon H = logγ( (1−γ)) maxs,a |R(s,a)| (Jie and Abbeel 2010).”
Guided Policy Search (2013, methodology): “Our approach follows prior methods that use importance sampling to address these challenges (Peshkin & Shelton, 2002; Kober & Peters, 2009; Tang & Abbeel, 2010).”
On the Role of Weight Sharing During Deep Option Learning (2019, background): “…scaling storage, these problems have been addressed in the literature by leveraging importance sampling (Precup, 2000b; Peshkin & Shelton, 2002; Jie & Abbeel, 2010; Mahmood et al., 2014) and scalable storage mechanisms (Robins, 1995; Li & Hoiem, 2016; Riemer et al., 2016, 2017; Shin et al.,…”