Learning from Scarce Experience

By Leonid Peshkin, Christian R. Shelton. 2002

75 citations recorded by the source corpus.

Searching the space of policies directly for the optimal policy has been one popular method for solving partially observable reinforcement learning problems. Typically, with each change of the target policy, its value is estimated from the results of following that very policy. This requires a large number of interactions with the environment as different polices are considered. We present a family of algorithms based on likelihood ratio estimation that use data gathered when executing one policy (or collection of policies) to estimate the value of a different policy. The algorithms combine estimation and optimization stages. The former utilizes experience to build a non-parametric representation of an optimized function. The latter performs optimization on this estimate. We show positive empirical results and provide the sample complexity bound.

Subfields: Artificial Intelligence, Management Science and Operations Research, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Meta & Continual Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “)}i=1 are n trajectory samples from πβ(τ) (Precup, 2000; Precup et al., 2001; Peshkin and Shelton, 2002).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Earlier works [51, 49, 52] used per-action importance sampling on Monte-Carlo returns to obtain an OPE return estimator.”
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, neutral): “Asides from these more recent works, the use of off-policy samples with policy gradients has been a popular direction of research (Peshkin & Shelton, 2002; Jie & Abbeel, 2010; Degris et al., 2012; Levine & Koltun, 2013).”
More Robust Doubly Robust Off-policy Evaluation (2018, methodology): “We can extend our MRDR results to the case that there are more than one behavior policy by replacing the IS part of our estimator with fused importance sampling (Peshkin & Shelton, 2002).”
P3O: Policy-on Policy-off Policy Optimization (2019, neutral): “Effective sample size is popularly used to measure the quality of importance sampling and to restrict the search space for parameter updates (Jie and Abbeel, 2010; Peshkin and Shelton, 2002).”
Offline Evaluation of Online Reinforcement Learning Algorithms (2016, methodology): “Importance sampling is widely-used in off-policy learning, as an objective function when using policy gradient methods (Levine and Koltun 2013; Peshkin and Shelton 2002) or as a way to re-weight samples in off-policy TD-learning methods (Mahmood, van Hasselt, and Sutton 2014; Sutton, Mahmood, and Wh”
On the Role of Weight Sharing During Deep Option Learning (2019, background): “…over time and issues with scaling storage, these problems have been addressed in the literature by leveraging importance sampling (Precup, 2000b; Peshkin & Shelton, 2002; Jie & Abbeel, 2010; Mahmood et al., 2014) and scalable storage mechanisms (Robins, 1995; Li & Hoiem, 2016; Riemer et al.,…”
Real-time reinforcement learning by sequential Actor-Critics and experience replay (2009, methodology): “(12) In order to construct an estimator of D(θ), we may apply the estimation technique called importance sampling (Rubinstein, 1981) frequently applied in RL (Peshkin & Shelton, 2002; Uchibe & Doya, 2004; Frank et al., 2008).”
Efficient Sample Reuse in EM-Based Policy Search (2009, background): “While plain importance sampling techniques have also been applied in the direct policy search, they were shown to be unstable [13, 7].”
Policy-Gradient Algorithms for Partially Observable Markov Decision Processes (2003, methodology): “IS can be used to implement off-policy policy-gradient methods [Peshkin and Shelton, 2002], reducing the number of world interactions required as described in Section 2.”
Offline policy evaluation across representations with applications to educational games (2014, methodology): “[19] proposes importance sampling for learning the value function given an MDP state space, and Peshkin and Shelton [18] investigate using IS to guide a policy gradient algorithm.”
On a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient (2010, methodology): “three components, which is equivalent to Peshkin and Shelton’s algorithm [10], which we label PS.”
A Generalization Error for Q-Learning (2005, methodology): “From the work of Kearns, Mansour, and Ng (1999, 2000) and Peshkin and Shelton (2002), we expect and find as well here that the number of trajectories needed to guarantee a specified error level is exponential in the horizon time, T .”
A comparative study of counterfactual estimators (2017, methodology): “We first show that, in this context, BIS is dominated by another estimator called Fused Importance Sampling (FIS, Peshkin & Shelton (2002)).”
Guided Policy Search (2013, methodology): “Prior work proposed estimating E[J(θ)] with importance sampling (Peshkin & Shelton, 2002; Tang & Abbeel, 2010).”
Reinforcement Learning Via Practice and Critique Advice (2010, unknown): “To compute U (θ, T ) it is straightforward to use likelihood weighting in order to estimate the utility of π θ using off-policy trajectories T , an approach that has been previously considered for policy gradient RL (Peshkin and Shelton 2002).”
Planning in Discrete and Continuous Markov Decision Processes by Probabilistic Programming (2015, methodology): “The closest related works include [20,21,24,25], they use importance sampling to evaluate a policy from samples generated with another policy.”