Batch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories

By Raphael Fonteneau, Susan A. Murphy, Louis Wehenkel, Damien Ernst. 2012

81 citations recorded by the source corpus.

In this paper, we consider the batch mode reinforcement learning setting, where the central problem is to learn from a sample of trajectories a policy that satisfies or optimizes a performance criterion. We focus on the continuous state space case for which usual resolution schemes rely on function approximators either to represent the underlying control problem or to represent its value function. As an alternative to the use of function approximators, we rely on the synthesis of “artificial trajectories” from the given sample of trajectories, and show that this idea opens new avenues for designing and analyzing algorithms for batch mode reinforcement learning.

Subfields: Artificial Intelligence, Control and Systems Engineering, Molecular Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Molecular Biology

In Reinforcement Learning

Citations in the corpus

Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, background): “(Alternatively, one could generate artiﬁcial trajectories for evaluation without explicitly referring to a model (Fonteneau et al., 2013).)”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, background): “One approach is to build a simulator that mimics the reward and next-state transitions in the real world, and then evaluate the target policy against the simulator [7, 14].”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, background): “The first approach is to build a simulator that mimics the reward and next-state transitions of the real environment (e.g., Fonteneau et al., 2013).”
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections (2019, background): “The problem of off-policy policy evaluation has been heavily studied in contextual bandits [12, 49, 52] and in the more general RL setting [14, 21, 26, 29, 34, 36, 37, 50, 51].”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, background): “, in health [18] and robotics [24] applications.”
Offline Evaluation of Online Reinforcement Learning Algorithms (2016, methodology): “Our methods do bear a relationship to off-policy learning work which has evaluated policies by synthesizing artiﬁcial trajectories (Fonteneau et al. 2010; 2013).”
Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation (2020, methodology): “…to tackle MDPs with inﬁnite or continuous state spaces, one often needs various forms of function approximation, and many methods like ﬁtted Q-iteration and least square policy iteration were developed (Jong & Stone, 2007; Lagoudakis & Parr, 2003; Grunewalder et al., 2012; Fonteneau et al., 2013).”
Doubly Robust Off-policy Evaluation for Reinforcement Learning (2015, background): “(2002) and variants that generates approximate models by artificial trajectories (Fonteneau et al., 2013).”
An Introduction to Deep Reinforcement Learning (2018, unknown): “One possibility is to use a policy evaluation step obtained by generating artificial trajectories from the data, without explicitly referring to a model, thus designing a Model-free Monte Carlo-like (MFMC) estimator (Fonteneau et al., 2013).”
Off-Policy Evaluation via the Regularized Lagrangian (2020, background): “While some approaches are model-based [10], or work by estimating the value function [8], most rely on importance reweighting to transform the off-policy data distribution to the on-policy target distribution.”
Harnessing Inﬁnite-Horizon Off-Policy Evaluation: Double Robustness via Duality (2019, background): “…value evaluation has been studied in contextual bandits (Dudík et al., 2011; Wang et al., 2017) and more general ﬁnite horizon RL settings (Fonteneau et al., 2013; Li et al., 2015; Jiang & Li, 2016; Thomas & Brunskill, 2016; Liu et al., 2018b; Farajtabar et al., 2018; Hanna et al., 2019;…”
Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon Reinforcement Learning? (2020, methodology): “Interestingly, this technique is related to the method proposed in Fonteneau et al. [2013] for policy evaluation, though there is no exploration component in their paper.”