Projected equation methods for approximate solution of large linear systems

By Dimitri P. Bertsekas, Huizhen Yu. 2009

85 citations recorded by the source corpus.

Subfields: Management Science and Operations Research, Statistics and Probability, Numerical Methods & Scientific Computing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

In Numerical Methods & Scientific Computing

Citations in the corpus

Minimax Weight and Q-Function Learning for Off-Policy Evaluation (2019, methodology): “…to LSTDQ in the linear setting, it is worth pointing out that the method of Liu et al. (2018)—which we will call MSWL (minimax state weight learning) for distinction and easy reference—corresponds to off-policy LSTD (Bertsekas and Yu, 2009; Dann et al., 2014); see Appendix A.5 for details.”
Emphatic TD Bellman Operator is a Contraction (2015, background): “The only exception that we are aware of, is a contraction-based argument for importance-sampling based LSTD, under the restrictive assumption that the behavior and target policies are very similar (Bertsekas & Yu, 2009).”
On Convergence of Emphatic Temporal-Difference Learning (2015, background): “For regular off-policy LSTD(λ) and TD(λ) (Bertsekas and Yu, 2009), it has been shown by Yu (2012) that the associated joint process of states and trace iterates exhibit useful properties, by which convergence results for LSTD(λ) can be derived.”
Off-policy learning with eligibility traces: a survey (2013, background): “The proof of Theorem 3 follows the general idea of that of Proposition 4 of Bertsekas and Yu (2009b).”
Least Squares Temporal Difference Methods: An Analysis under General Conditions (2012, methodology): “The line of our analysis is considerably diﬀerent from those in the literature for similar on-policy TD methods (Tsitsiklis and Van Roy [34], Nedi´c and Bertsekas [24]) and oﬀ-policy TD methods [25, 8].”
Scaling Up Robust MDPs by Reinforcement Learning (2013, background): “We note that a similar difﬁculty arises in off-policy RL [5, 18] (in fact, our Assumption 2 is similar to an assumption in [5]) , where some algorithms are shown to converge to a solution of (3) even when Π T π is not a contraction [20, 18].”
Stabilization of Stochastic Iterative Methods for Singular and Nearly Singular Linear Systems (2014, methodology): “Also, the sample sequence can be generated through state transitions of an irreducible Markov chain as, for example, in temporal difference methods for cost evaluation problems in ADP (see Bradtke and Barto [12], Boyan [11], Nedić and Bertsekas [40], Bertsekas [4]) or for general projected equations”
Policy evaluation with temporal differences: a survey and comparison (2015, background): “For the sample-based implementation, we arrive at the off-policy parameter estimates of LSTD proposed by Bertsekas and Yu (2009) At = t∑ i=0 φi[φi − γρiφi+1]T , bt = t∑ i=0 φiρiri.”
Approximate policy iteration: a survey and some new methods (2011, methodology): “b) In Bertsekas and Yu [24], assuming that λ max (i,j) aij”
Weak Convergence Properties of Constrained Emphatic Temporal-difference Learning with Constant and Slowly Diminishing Stepsize (2015, methodology): “, (16)It is not necessary to multiply the term φ(St)θt by ρt, and that version of the algorithm was the one given in [5, 51].”
Q-learning and enhanced policy iteration in discounted dynamic programming (2010, background): “For example, some LSTD()type policy evaluation algorithms use the richer set of data thus obtained together with importance sampling to guarantee convergence (Bertsekas and Yu [13], Yu [50, 51]).”
Reinforcement Learning under Model Mismatch (2017, methodology): “Under a technical assumption similar to [5, 24] we show convergence of robust approximate policy iteration and value iteration algorithms for linear architectures.”
Scaling Up Robust MDPs using Function Approximation (2014, result): “We note that a similar difficulty arises in off-policy RL (Bertsekas & Yu, 2009; Sutton et al., 2009), and our Assumption 2 is in fact similar to an assumption of Bertsekas & Yu 2009.”
Convergence of Least Squares Temporal Difference Methods Under General Conditions (2010, methodology): “The off-policy LSTD(λ) algorithm we will analyze was first given by (Bertsekas & Yu, 2009) in the general context of approximating solutions of linear systems of equations.”