Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path

By András Antos, Csaba Szepesvari, R´emi Munos. 2006

509 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistics and Probability, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, neutral): “This issue arises in Q-learning as well, and others have proposed possible remedies which can also be applied to PCL [2].”
Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “To avoid the cost of the double sampling, a common approach is to instead minimize the Bellman residual (Lagoudakis & Parr, 2003; Antos et al., 2008): (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) The Bellman residual is the sum of the expression in Eq.”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, methodology): “However, as discussed in the literature (Antos et al., 2008; Lazaric et al., 2012; Dai et al., 2018; Nachum et al., 2019), using the blocking technique of Yu (1994), the statistical error provided here can be generalized to β-mixing samples in a single sample path.”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, background): “See Antos et al. (2008); Dai et al.”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “There are several alternatives to handle this issue (e.g., Antos et al., 2008; Farahmand et al., 2016; Feng et al., 2019).”
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections (2019, methodology): “However, by exploiting blocking technique in Proposition 15 of [53], following [2, 23, 9], all the sample complexity we provided can be easily generalized for single β -mixing sample path, i.e. , { s i , a i , r i , s i } =1 is strictly stationary and mixing in an exponential rate with parameter b,…”
Learning When-to-Treat Policies (2019, background): “…is typically referred to as a “finite horizon” setting, and there is also a large literature on the “infinite horizon” setting where T is not bounded (see, e.g., Antos, Szepesvári, and Munos 2008a, 2008b; Munos and Szepesvári 2008; Q. Liu et al. 2018; Uehara and Jiang 2019; Luckett et al. 2020).”
Information-Theoretic Considerations in Batch Reinforcement Learning (2019, methodology): “…(2012); Farahmand et al. (2016)), according to our knowledge, fast rate for the minimax al-gorithm has not been established before: for example, Antos et al. (2008); Munos & Szepesv´ari (2008) obtain an error rate of n − 1 / 4 in closely related settings, but their rates do not improve to n −…”
Off-policy learning with eligibility traces: a survey (2013, methodology): “Such a bias adds an uncontrolled variance term to Ā and b̄ (for instance, see Antos et al. (2006)); an interesting consequence is that Ã is always non-singular10.”
Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes (2019, methodology): “, 2007) or estimate the q-function directly using fitted LSTDQ or more general q-iteration (Antos et al., 2008; Lagoudakis and Parr, 2004; Le et al., 2019) (we further review estimation of q-functions in Section 4.”
Batch Policy Learning under Constraints (2019, methodology): “We perform grid search over a range of regularization parameters λ for both Least-Squares Policy Iteration - LSPI ((Lagoudakis & Parr, 2003)) and Fitted Q Iteration - FQI ((Ernst et al., 2005)).”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, methodology): “In value based RL, the core of most prominent approaches is to learn the Qfunction through minimizing a surrogate to Bellman residual (Schweitzer and Seidmann, 1985; Lagoudakis and Parr, 2003; Antos et al., 2008) using temporal difference (TD) update (Tesauro, 1995).”
Finite-Sample Analysis of Proximal Gradient TD Algorithms (2015, methodology): “…in RL has been focused on batch RL (or approximate dynamic programming) algorithms (e.g., Kakade and Langford 2002; Munos and Szepesv´ari 2008; Antos et al. 2008; Lazaric et al. 2010a), especially those that are least squares TD (LSTD)-based (e.g., Lazaric et al. 2010b; Ghavamzadeh et al.…”
Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view (2010, methodology): “As pointed out by Antos et al. (2008); Farahmand et al.”
Contextual Decision Processes with low Bellman rank are PAC-Learnable (2016, background): “This is weaker than the realizability assumption made in the value-based RL literature, that F contains the optimal Q-value function of an MDP Q (cid:63) [Antos et al., 2008, Krishnamurthy et al., 2016].”
A Kernel Loss for Solving the Bellman Equation (2019, methodology): “Being fixed-point iteration methods, FVI-style algorithms do not optimize any objective function, and their convergence is guaranteed only in rather restricted cases (e.g., Gordon, 1995; Tsitsiklis & Van Roy, 1997; Antos et al., 2008).”