Fitted Q-iteration by Advantage Weighted Regression

By Gerhard Neumann, Jan Peters. 2008

79 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “Neumann & Peters (2009) proposed LAWER, a kernel-based ﬁtted Q-iteration algorithm where the Bellman error is weighted by the normalized advantage of each state-action pair.”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “pared to Vecerík et al. [43] and Nair et al. [27].ˇ Our method builds on algorithms that implement a maximum likelihood objective for the actor, based on an expectation-maximization formulation of RL [31, 29, 40, 34, 30, 1]. Most closely related to our method in this respect are the algorithms propo”
Critic Regularized Regression (2020, methodology): “…re-weighted imitation learning (MARWIL) [38], best-action imitation learning (BAIL) [7], advantage-weighted behavior models (ABM) [31] and advantage weighted regression [28], which has previously been studied in the form of a Fitted Q-iteration algorithm with low-dimensional policy classes [27].”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “A number of works have leveraged this insight to develop RL algorithms [3, 29, 31, 32].”
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (2019, background): “In this regard V-MPO can also be compared to Fitted Q -iteration by Advantage Weighted Regression (Neumann & Peters, 2009), which learns a Q -function but uses only one action per state.”
Adaptive Control of Epileptic Seizures using Reinforcement Learning (2010, background): “It should be mentioned that there exists a considerable body of work on RL with continuous actions that has to deal with generalization across actions [44, 40, 50], but a full review of that material is outside the scope of this thesis.”
Online Model Learning Algorithms for Actor-Critic Control (2015, methodology): “Memory-based learning methods have successfully been applied to RL before, mostly as an approximator for the value function (Gabel and Riedmiller, 2005; Neumann and Peters, 2009) and in some cases also for the process model (Forbes and Andre, 2002; Ng et al.”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, background): “However, in contrast with our work, the results from Kumar et al. [26] suggest that good performance cannot be achieved through RvS learning but only through additional advantage-weighting [32, 34].”
A Survey on Policy Search for Robotics (2013, methodology): “This parameter is either specified by the user [39] or can be set by heuristics, for example, setting β to a multiple of the standard deviation of the rewards [49, 50].”
Hierarchical Relative Entropy Policy Search (2014, background): “, 2003), motion templates (Neumann and Peters, 2009) or parametrized skills (Da Silva et al.”
Sample-based informationl-theoretic stochastic optimal control (2014, methodology): “While in [5] a strongly related formulation has been used to optimize parameters of high-level decisions where an episode consisted only of a small number of decisions, we want to cope with large time horizons of H = 50 to H = 100.”
Learning to Reach Goals via Iterated Supervised Learning (2019, background): “…choose a small subset of trajectories to imitate [Oh et al., 2018, Hao et al., 2019] or learn a separate value function to reweight past experience [Neumann and Peters, 2009, Abdolmaleki et al., 2018, Peng et al., 2019], we show that GCSL learns efﬁciently while training on every previous…”
Learning Agile Robotic Locomotion Skills by Imitating Animals (2020, methodology): “To identify z ∗ , we use advantage-weighted regression (AWR) [40, 46], a simple off-policy RL algorithm.”