Finite-sample Analysis of Bellman Residual Minimization

By Odalric-Ambrym Maillard, R´emi Munos, Alessandro Lazaric, Mohammad Ghavamzadeh. 2010

34 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Information-Theoretic Considerations in Batch Reinforcement Learning (2019, methodology): “(1)In this paper, we restrict ourselves to the so-called one-path setting and do not allow multiple samples from the same state (Sutton & Barto, 1998; Maillard et al., 2010), which is only feasible in certain simulated environments and allows algorithms to succeed with realizability as the only repr”
Batch Policy Learning under Constraints (2019, methodology): “…F, when Tg /∈ F, a notion of distance, known as inherent Bellman error was first proposed by (Munos, 2003) and used in the analysis of related ADP algorithms (Munos & Szepesvári, 2008; Munos, 2007; Antos et al., 2008a;b; Lazaric et al., 2010; 2012; Lazaric & Restelli, 2011; Maillard et al., 2010).”
On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes (2012, background): “Interestingly, this very constant 2γ (1−γ)2 appears in many works analyzing AVI algorithms [25, 11, 27, 12, 13, 23, 7, 6, 20, 21, 22, 9], API algorithms [15, 19, 16, 1, 8, 18, 5, 17, 10, 3, 9, 2] and in one of their generalization [26], suggesting that it cannot be improved.”
Transfer in Reinforcement Learning: A Framework and a Survey (2012, background): “…as fitted value iteration (Munos and Szepesvári, 2008), LSTD (Farahmand et al, 2008; Lazaric et al, 2010), and Bellman residual minimization (Antos et al, 2008; Maillard et al, 2010), at the best of our knowledge, at the moment there is no finite-sample analysis for any transfer algorithm in RL.”
Reinforcement Learning in Continuous State and Action Spaces (2012, methodology): “Using (δt)2 as a sample for this error (withB = Bπ ) leads to a biased estimate, but other approaches have been proposed that use this error (Antos et al, 2008b; Maillard et al, 2010).”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, methodology): “(2017) has proposed a batch RL method to find an approximate Nash equilibrium via Bellman residue minimization (Maillard et al., 2010).”