Q-Learning: Theory and Applications
By Jesse Clifton, Eric B. Laber. 2020
460 citations recorded by the source corpus.
Q-learning, originally an incremental algorithm for estimating an optimal decision strategy in an infinite-horizon decision problem, now refers to a general class of reinforcement learning methods widely used in statistics and artificial intelligence. In the context of personalized medicine, finite-horizon Q-learning is the workhorse for estimating optimal treatment strategies, known as treatment regimes. Infinite-horizon Q-learning is also increasingly relevant in the growing field of mobile health. In computer science, Q-learning methods have achieved remarkable performance in domains such as game-playing and robotics. In this article, we ( a) review the history of Q-learning in computer science and statistics, ( b) formalize finite-horizon Q-learning within the potential outcomes framework and discuss the inferential difficulties for which it is infamous, and ( c) review variants of infinite-horizon Q-learning and the exploration-exploitation problem, which arises in decision problems with a long time horizon. We close by discussing issues arising with the use of Q-learning in practice, including arguments for combining Q-learning with direct-search methods; sample size considerations for sequential, multiple assignment randomized trials; and possibilities for combining Q-learning with model-based methods.
Subfields: Management Science and Operations Research, Statistics and Probability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Management Science and Operations Research
- Stefan Wager
- Xinkun Nie
- Emma Brunskill
- Susan Athey
- Nathan Kallus
- Ian Osband
- Brendan O'Donoghue
- Lihong Li
In Statistics and Probability
- Stefan Wager
- Xinkun Nie
- Emma Brunskill
- Susan Athey
- Edward H. Kennedy
- Mihaela van der Schaar
- Liangyuan Hu
- Guido W. Imbens
Citations in the corpus
- Learning When-to-Treat Policies (2019, methodology): “As this is a large literature spanning several fields, a comprehensive overview of the literature on dynamic decision rules is beyond the scope of this article; instead, we refer the reader to recent review papers (Vansteelandt and Joffe 2014; Kosorok and Laber 2019; Clifton and Laber 2020; Levine e”