Approximate solutions to markov decision processes

By Geoffrey J. Gordon, Tom Michael Mitchell. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Reinforcement learning in robotics: A survey (2013, background): “Similarly, it is well known that linear value-function approximation is unstable in the offpolicy case (Tsitsiklis and Van Roy, 1997; Sutton and Barto, 1998; Gordon, 1999).”
Batch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories (2012, background): “The same behavior could probably be observed by using other algorithms that combine dynamic programming strategies with kernel-based approximators or averagers (Boyan and Moore 1995; Gordon 1999; Ormoneit and Sen 2002).”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “d [72]. In practice, using function approximators in RL requires making crucial representational decisions and poor design choices can result in estimates that diverge from the optimal value function [73, 69, 74, 75, 76, 77]. In particular, function approximation, bootstrapping, and o-policy learnin”
Tree-Based Batch Mode Reinforcement Learning (2005, methodology): “They are collectively referred to as kernel-based methods (see Gordon, 1999; Ormoneit and Sen, 2002).”
Stability and generalization (2002, background): “In order to do this, we need some standard definitions about convex functions which we deferred to Appendix C where most of the material can be found in Rockafellar (1970) and in Gordon (1999). Lemma 21 Under the conditions of Lemma 20, when F is a vector space and N is a proper closed convex functi”
Adaptive Treatment of Epilepsy via Batch-mode Reinforcement Learning (2008, methodology): “The ﬁtted Q iteration algorithm (Ernst, Geurts, & We-henkel 2005), which builds on earlier work on ﬁtted value iteration (Gordon 1999; Ormoneit & Sen 2002), takes as input a set F of 4-tuples of the form < s t , a t , r t , s t +1 > , where each tuple is an example of the one-step transition…”
Stability and Generalization (2002, background): “In order to do this, we need some standard definitions about convex functions which we deferred to Appendix C where most of the material can be found in Rockafellar (1970) and in Gordon (1999). Lemma 21 Under the conditions of Lemma 20, when F is a vector space and N is a proper closed convex functi”
Randomized Value Functions via Multiplicative Normalizing Flows (2018, background): “The action-value function of the optimal policy is Q (cid:63) ( s, a ) = arg max π Q π ( s, a ) and it satisﬁes the Bellman optimality equation: Fitted Q iteration (FQI) (Gordon, 1999; Riedmiller, 2005) assumes that the entire learning dataset of agent interactions is available from the start.”
A Unified View of Matrix Factorization Models (2008, background): “For a closed, proper, convex function F : Rm×n → R the generalized Bregman divergence [7,8] between matrices Θ and X is”
Reinforcement Learning based Recommender Systems: A Survey (2021, methodology): “Fitted Q algorithm is an approximate method that is inspired by the idea of itted value iteration proposed by Gordon [63].”
Reinforcement Learning: An Introduction (1998, methodology): “It is a variant of fitted value iteration (Gordon, 1999) adapted to Q-learning.”
What are the Statistical Limits of Offline RL with Linear Function Approximation? (2020, background): “For example, when running on oﬀ-policy data (sometimes termed as “experience replay”), deep Q -networks (DQN) [Mnih et al., 2015] can be viewed as an analog of Fitted Q -Iteration [Gordon, 1999] with neural networks being the function approximators.”
Kernel-Based Reinforcement Learning (2004, background): “In the context of reinforcement learning, local averaging has been suggested in work by Rust (1997) and Gordon (1999), making the assumption that the transition probabilities of the MDP are known and can be used for learning.”
Reinforcement Learning in Continuous State and Action Spaces (2012, background): “(10) This is the conventional TD learning update and it usually converges faster than the residual-gradient update (Gordon, 1995, 1999).”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, methodology): “…policy completeness assumption with regards to a linear feature mapping 1 along with data coverage assumption, then Fitted-Q iteration (FQI) [Gordon, 1999] — a classical oﬄine Bellman backup based method — can provably evaluate a policy with low sample complexity (in the dimension of the…”
Effective reinforcement learning for mobile robots (2002, methodology): “The algorithm is based on the observation by Gordon[ 7 ]thatafunctionapproximatorcansafelybe used to replace the tabular value function representation if it never extrapolates from its training data.”