Adaptive importance sampling for value function approximation in off-policy reinforcement learning

By Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiyama, Jan Peters. 2009

51 citations recorded by the source corpus.

Subfields: Automotive Engineering, Cardiology and Cardiovascular Medicine, Control and Systems Engineering, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

1. Sergey Levine
2. Aviral Kumar
3. Emma Brunskill
4. Hado van Hasselt
5. Pieter Abbeel
6. Ofir Nachum
7. Shixiang Shane Gu
8. Michael Janner

In Cardiology and Cardiovascular Medicine

1. Sergey Levine

In Control and Systems Engineering

1. Sergey Levine
2. Aviral Kumar
3. Xue Bin Peng
4. Pieter Abbeel
5. Chelsea Finn
6. Takafumi Kanamori
7. Taiji Suzuki
8. Emma Brunskill

In Reinforcement Learning

1. Sergey Levine
2. Aviral Kumar
3. Emma Brunskill
4. Ofir Nachum
5. Pieter Abbeel
6. Chelsea Finn
7. Shixiang Shane Gu
8. George Tucker

In Optimization & Training

1. Hado van Hasselt
2. A. Mahmood
3. Aviral Kumar
4. Sergey Levine
5. Emma Brunskill
6. Xue Bin Peng
7. Zoran Popovic
8. Travis Mandel

Citations in the corpus

• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “Importance sampling is a simple strategy for incorporating off-policy data (Sutton & Barto, 1998; Meuleau et al., 2000; Hachiya et al., 2009), but can introduce optimization instabilities due to the potentially large variance of the importance sampling estimator.”
• Least-squares two-sample test (2011, methodology): “When learning is performed under non-stationary environment, e.g., in brain-computer interface (Sugiyama et al., 2007) and robot control (Hachiya et al., 2009), testing homogeneity of data generating distributions allows one to determine whether some adaptation scheme should be used or not.”
• Weighted importance sampling for off-policy learning with linear function approximation (2014, methodology): “The Adaptive Per-Decision Importance Weighting (APDIW) method by Hachiya et al. (2009) is superficially similar to WIS-LSTD(), there are several important differences.”
• Offline policy evaluation across representations with applications to educational games (2014, methodology): “[9] use a cross-validated importance sampling estimate of value function error (as estimated by the Bellman residual) to determine the best value of a single parameter of a fixed policy representation.”
• Model-Based Policy Gradients with Parameter-Based Exploration by Least-Squares Conditional Density Estimation (2013, methodology): “So far, various machine learning techniques have been employed for better value function approximation, such as least-squares approximation [12], manifold learning [17], efficient sample reuse [6], active learning [2], and robust learning [16].”