Fitted Q-iteration in continuous action-space MDPs

By András Antos, R´emi Munos, Csaba Szepesvari. 2007

260 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning (2019, methodology): “When the sample space and action space are continuous, we can apply some functional approximation to q-function modeling and use the temporal-difference method (Lagoudakis and Parr, 2004) or fitted Q–iteration (Antos et al., 2008).”
Learning When-to-Treat Policies (2019, background): “Finally, we note that while we focus on the finite horizon setting, there is a large literature in policy evaluation in the infinite horizon setting (see Antos et al. (2008a,b); Liu et al. (2018a); Luckett et al.”
Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection (2016, methodology): “Repeatedly deploying the latest grasp network g( It, vt), collecting additional data, and refitting g( It, vt) can then be regarded as fitted Q iteration (Antos et al., 2008).”
Batch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories (2012, methodology): “Extensions of the FQI algorithm to continuous action spaces have also been proposed (Antos et al. 2007).”
Batch Policy Learning under Constraints (2019, methodology): “We perform grid search over a range of regularization parameters λ for both Least-Squares Policy Iteration - LSPI ((Lagoudakis & Parr, 2003)) and Fitted Q Iteration - FQI ((Ernst et al., 2005)).”
Dynamic policy programming (2010, methodology): “Another interesting line of future research is to devise finite-sample PAC bounds for SADPP in the spirit of previous theoretical results available for fitted value iteration and fitted Q-iteration (Munos and Szepesvári, 2008; Antos et al., 2008; Munos, 2005).”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, methodology): “For ﬁnite-horizon non-discounted MDPs, the value V or action-conditional value Q of a policy π is which may be extended to regularize the policy’s entropy with some temperature α ≥ 0 as The value function of a given policy can be estimated ( i.e. policy evaluation ) by explicitly rolling out the wor”
Sample-Optimal Parametric Q-Learning Using Linearly Additive Features (2019, methodology): “Fitted value iteration (VI) applies to more general function approximators of the value function Munos and Szepesv´ari [2008], Antos et al. [2008a], Farahmand et al. [2010], Antos et al. [2008b], where v is approximated within a low-dimensional function space F .”
Adaptive Control of Epileptic Seizures using Reinforcement Learning (2010, background): “In terms of the theoretical quality of the solution obtained by the algorithm, some finite-sample performance bounds for FQI were proven in [2] and [47] for a particular class of regressors.”
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, background): “, maxπ C [26], [33], [57]–[61] and some definitions are more complex and stronger assuming a bounded ratio per time step [24], [25], [62]–[64].”
Tight Performance Bounds for Approximate Modified Policy Iteration with Non-Stationary Policies (2013, background): “…We use the fact that γTK < 1−γ ⇔ T > log (1−γ)K log 1 γ ' log (1−γ)K 1−γ with K = maxs,a |r(s,a)| 1−γ . scheme—see for instance (Bertsekas and Tsitsiklis, 1996; Antos et al., 2007a,b; Scherrer et al., 2012a)— and thus controlled through some weighted quadratic L2,µ norm, defined as ‖f‖2,µ = √∫…”
Convergent Fitted Value Iteration with Linear Function Approximation (2011, result): “This is in contrast to previous work that explores how properties of the underlying MDP and properties of the function approximation space jointly influence convergence of the algorithm [1, 14, 6].”
Reinforcement Learning in Continuous State and Action Spaces (2012, methodology): “Using (δt)2 as a sample for this error (withB = Bπ ) leads to a biased estimate, but other approaches have been proposed that use this error (Antos et al, 2008b; Maillard et al, 2010).”
Fitted Q-iteration by Advantage Weighted Regression (2008, background): “We consider the use of continuous actions for fitted Q-iteration (FQI) based algorithms.”
Toward Off-Policy Learning Control with Function Approximation (2010, methodology): “…of solving the latent learning problem include using non-incremental methods that are more computationally expensive (e.g., Lagoudakis & Parr, 2003; Antos et al., 2008; 2007), possibly with nonlinear value function approximation methods (e.g., Antos et al., 2008; 2007); putting restrictions on the…”
Near-Optimal Offline Reinforcement Learning via Double Variance Reduction (2021, methodology): “The study of offline policy optimization can be dated back to the classical Fitted Q-Iteration algorithm (Antos et al., 2008a,b). The sample complexity for offline policy optimization is studied in a line of recent work (Chen & Jiang, 2019; Le et al., 2019; Xie & Jiang, 2020b,a; Liu et al., 2020b). ”