Online Off-policy Prediction
By Sina Ghiassian, Andrew Patterson, Martha White, Richard S. Sutton, Adam White. 2018
23 citations recorded by the source corpus.
This paper investigates the problem of online prediction learning, where learning proceeds continuously as the agent interacts with an environment. The predictions made by the agent are contingent on a particular way of behaving, represented as a value function. However, the behavior used to select actions and generate the behavior data might be different from the one used to define the predictions, and thus the samples are generated off-policy. The ability to learn behavior-contingent predictions online and off-policy has long been advocated as a key capability of predictive-knowledge learning systems but remained an open algorithmic challenge for decades. The issue lies with the temporal difference (TD) learning update at the heart of most prediction algorithms: combining bootstrapping, off-policy sampling and function approximation may cause the value estimate to diverge. A breakthrough came with the development of a new objective function that admitted stochastic gradient descent variants of TD. Since then, many sound online off-policy prediction algorithms have been developed, but there has been limited empirical work investigating the relative merits of all the variants. This paper aims to fill these empirical gaps and provide clarity on the key ideas behind each method. We summarize the large body of literature on off-policy learning, focusing on 1- methods that use computation linear in the number of features and are convergent under off-policy sampling, and 2- other methods which have proven useful with non-fixed, nonlinear function approximation. We provide an empirical study of off-policy prediction methods in two challenging microworlds. We report each method's parameter sensitivity, empirical convergence rate, and final performance, providing new insights that should enable practitioners to successfully extend these new methods to large-scale applications.[Abridged abstract]
Subfields: Artificial Intelligence, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Shimon Whiteson
- Martha White
- Shangtong Zhang
- Adam White
- Wendelin Böhmer
- Richard S. Sutton
- Sergey Levine
- A. Mahmood
In Management Science and Operations Research
- Martha White
- Shimon Whiteson
- Shangtong Zhang
- Adam White
- Wendelin Böhmer
- Richard S. Sutton
- A. Mahmood
- Hado van Hasselt
Citations in the corpus
- Generalized Off-Policy Actor-Critic (2019, background): “This objective is referred to as the excursion objective (White, 2018; Ghiassian et al., 2018), as it corresponds to the excursion setting (Sutton et al.”
- Planning with Expectation Models (2019, methodology): “We evaluate the proposed method for the non-linear model choice in two simple yet illustrative domains: stochastic variants of Four Rooms [Sutton et al. , 1999; Ghiassian et al. , 2018] and Mountain Car [Sutton, 1996].”