Kernel-Based Reinforcement Learning

By Guanghua Hu, Yuqin Qiu, Liming Xiang. 2006

426 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “After sufﬁcient iterations KBRL will converge correctly to Q ( s 0 , a 1 ) = 1 1 − γ 2 , Q ( s 1 , a 0 ) = γ 1 − γ 2 .”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, methodology): “Model Construction We apply Kernel-based Reinforcement Learning (Ormoneit & Sen, 2002) for the reason that in Sail R min is rarely reached hence too pessimistic as a rough estimate of the magnitude of reward obtained per step.”
Modelling transition dynamics in MDPs with RKHS embeddings (2012, background): “Other related approaches include Rasmussen & Kuss (2003), Deisenroth et al. (2009), Ormoneit & Sen (1999) and Kroemer & Peters (2011).”
Increasing the Action Gap: New Operators for Reinforcement Learning (2015, background): “D ( x | z ) = I [ x = z ] , we recover the class of averagers (Gordon 1995; e.g., multilinear interpolation, illustrated in Figure 2) and kernel-based meth-ods (Ormoneit and Sen 2002).”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, background): “However, these traditional algorithms only come with convergence guarantees for non-parametric function approximation [Gordon, 1995, Ormoneit and Sen, 2002], have no guarantees on the quality of the learned policy, and scale poorly to high dimensional tasks.”
Batch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories (2012, methodology): “For example, Ormoneit and Sen have developed a batch mode RL algorithm in 2002 (Ormoneit and Sen 2002) using kernel approximators, for which theoretical analyses are also provided.”
Batch Policy Learning under Constraints (2019, background): “One can also consider modelbased alternatives (Ormoneit & Sen, 2002).”
Reinforcement Learning Strategies for Clinical Trials in Non-small Cell Lung Cancer (2011, methodology): “Besides the RF and ERT methods, other methodologies for fitting Q include, but are not limited to, neural networks, kernel-based regression (Ormoneit and Sen, 2002), and support vector machines (SVM) (Vapnik, 1995).”
A Kernel Loss for Solving the Bellman Equation (2019, methodology): “In RL, authors have used kernels either to smooth the estimates of transition probabilities and rewards (Ormoneit & Sen, 2002), or to represent the value function (e.g., Xu et al., 2005, 2007; Taylor & Parr, 2009).”
Learning to Drive a Real Car in 20 Minutes (2007, background): “The approach is fully data-driven, which means that no simulation model of the car and no pre-deﬁned control law is used for learning.”
SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation (2017, background): “…classes, such as those with a non-expansion property, some of the ﬁnite-state MDP theory continues to apply with modiﬁcations (Gordon, 1995; Ormoneit & Sen, 2002; Antos et al., 2008); • when linear value function approximation in certain cases, convergence is guaranteed: for evaluating a…”
Adaptive Control of Epileptic Seizures using Reinforcement Learning (2010, methodology): “3, relies on sample trajectories from the MDP to learn a policy and is known to be converge to the optimal value function asymptotically under some continuity assumption and for particular classes of approximation function [53].”
Approximate MaxEnt Inverse Optimal Control and Its Application for Mental Simulation of Human Interactions (2015, methodology): “The forth baseline is the kernel-based reinforcement learning (KRL) (Ormoneit and Sen 2002) value function approximation presented in (Huang and Kitani 2014), which applies kernel regression on a value function learned by MaxEnt IOC to get a continuous value function over the whole state space.”
Deep auto-encoder neural networks in reinforcement learning (2010, methodology): “When using a (kernel-based) averager [8], [7] for approximating the Q-function, the series of approximations { ˆ Q i } produced by the FQI algorithm—under some assumptions— is guaranteed to converge to a unique fix-point ¯ Q that is within a specific bound of the optimal Q-function Q * [13], [7].”
Reinforcement learning for control: Performance, stability, and deep approximators (2018, background): “Many RL techniques also use nonparametric approximators, which vary their shape and number of parameters as a function of the dataset, e.g. RL with kernel regression (Ormoneit and Sen, 2002; Farahmand et al., 2009), Gaussian processes (Engel et al., 2005), regression trees (Ernst et al., 2005), etc.”
Model-based function approximation in reinforcement learning (2007, methodology): “P a This version of AMBI can also be construed as an online implementation of Kernel-Based Reinforcement Learning [12] (KBRL), discussed in Section 6, combined with the exploration mechanism of Prioritized Sweeping.”
A Clustering-Based Graph Laplacian Framework for Value Function Approximation in Reinforcement Learning (2014, methodology): “One of the earliest works in kernel methods for RL was published in [31], where the state-value functions of MDPs were approximated by kernel-based locally weighted averaging.”
Bias-corrected Q-learning to control max-operator bias in Q-learning (2013, background): “The poor performance of Q-learning can be traced back to the bias caused by taking the maximum over actions having overestimated expected returns [17], [18], [19].”