Tree-Based Batch Mode Reinforcement Learning

By Damien Ernst, Pierre Geurts, Louis Wehenkel. 2005

1,281 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Basic offline dynamic programming algorithms can be constructed on the basis of fitted Q-learning methods (Ernst et al., 2005), as well as policy iteration methods (Sutton and Barto, 1998).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “Offline RL, also known as batch RL, offers an appealing alternative [11, 15, 30, 3, 27, 54, 34].”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “Furthermore, unlike DDPG and DQN, BCQ exhibits a highly stable value function in the presence of off-policy samples, suggesting extrapolation error has been successfully mitigated.”
Toward Interpretable-AI Policies Using Evolutionary Nonlinear Decision Trees for Discrete-Action Systems (2020, methodology): “In [27], regression trees are derived using classical methods, such as CART [22] and Kd-tree [28], to model Q -function through supervised training on batch of experiences and comparative study is made with ensemble techniques.”
Fitted Q-Learning for Relational Domains (2020, methodology): “…of classical approaches for this problem based on Approximate Dynamic Programming (ADP) is the ﬁtted value iteration algorithm (Boyan and Moore 1995; Ernst, Geurts, and Wehenkel 2005; Ried-miller 2005), a batch mode approximation scheme that employs function approximators in each iteration to…”
Reinforcement Learning in Healthcare: A Survey (2019, methodology): “vector regression (SVG) [104] and extremely randomized trees (ERT) [41], were applied to fit the approximated Q-functions to the generated trial data.”
Verifiable Reinforcement Learning via Policy Extraction (2018, methodology): “However, decision trees are challenging to learn even in the supervised setting; there has been some work learning decision tree policies for reinforcement learning [13], but we ﬁnd that they do not even scale to simple problems like cart-pole [5].”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “…with non-linear function approximators has mainly considered the “online” or “growing batch” settings, where additional exploration data is collected (Ernst et al., 2005; Riedmiller, 2005; Ormoneit & Sen, 2002); though some success for batch RL in discrete domains has been reported (Agarwal et…”
Minimax Weight and Q-Function Learning for Off-Policy Evaluation (2019, methodology): “An alternative but closely related approach is to fit Qπe directly from data using standard approximate dynamic programming (ADP) techniques, e.g., the policy evaluation analog of Fitted Q-Iteration (Ernst et al., 2005; Le et al., 2019).”
Safe Policy Improvement with Soft Baseline Bootstrapping (2019, background): “, 1999), technical process control (Ernst et al., 2005; Riedmiller, 2005), medical applications (Guez et al.”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “RL on a ﬁxed dataset without any online interactions with the environment has been long studied [16, 31, 52, 34, 55].”
Learning When-to-Treat Policies (2019, methodology): “One different but very influential line of work on dynamic decision rules uses Q-learning (Robins 1986; Watkins and Dayan 1992; Ernst, Geurts, and Wehenkel 2005; Murphy 2005) which, at a high level, seeks to solve a noisy dynamic programming problem to learn an optimal policy (see Clifton and Laber…”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, methodology): “The sampled variant corresponds to FQI as described in the literature (Ernst et al., 2005; Riedmiller, 2005), while the exact variant is analogous to conventional ADP (Bertsekas & Tsitsiklis, 1996).”
Safe Policy Improvement with Baseline Bootstrapping (2017, methodology): “This policy may be found indifferently using dynamic programming on the explicitly modelled MDP M̂ ,Q-learning with experience replay until convergence (Sutton & Barto, 1998), or Fitted-Q Iteration with a one-hot vector representation of the state space (Ernst et al., 2005).”
Informing sequential clinical decision-making through reinforcement learning: an empirical study (2010, methodology): “That is, we use linear function approximation to estimate the state-action value function (Lagoudakis and Parr 2003; Irodova and Sloan 2005; Ernst et al. 2005; Parr et al. 2008).”
Safe Policy Improvement with an Estimated Baseline Policy (2019, methodology): “Classically, batch RL algorithms apply dynamic programming on the samples in the dataset (Lagoudakis and Parr 2003; Ernst, Geurts, and Wehenkel 2005).”
Information-Theoretic Considerations in Batch Reinforcement Learning (2019, methodology): “A representative algorithm for this setting is Fitted Q-Iteration (FQI) (Ernst et al., 2005; Szepesv´ari, 2010).”
A General Family of Robust Stochastic Operators for Reinforcement Learning (2018, methodology): “It is important to note, as also emphasized in [3], that our approach can be extended to variants of the Bellman operator such as SARSA [10], policy evaluation [11] and ﬁtted Q -iteration [7].”