Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges

By Sofia S. Villar, J. Bowden, James M. S. Wason. 2015

377 citations recorded by the source corpus.

Multi-armed bandit problems (MABPs) are a special type of optimal control problem well suited to model resource allocation under uncertainty in a wide variety of contexts. Since the first publication of the optimal solution of the classic MABP by a dynamic index rule, the bandit literature quickly diversified and emerged as an active research topic. Across this literature, the use of bandit models to optimally design clinical trials became a typical motivating application, yet little of the resulting theory has ever been used in the actual design and analysis of clinical trials. To this end, we review two MABP decision-theoretic approaches to the optimal allocation of treatments in a clinical trial: the infinite-horizon Bayesian Bernoulli MABP and the finite-horizon variant. These models possess distinct theoretical properties and lead to separate allocation rules in a clinical trial design context. We evaluate their performance compared to other allocation rules, including fixed randomization. Our results indicate that bandit approaches offer significant advantages, in terms of assigning more patients to better treatments, and severe limitations, in terms of their resulting statistical power. We propose a novel bandit-based patient allocation rule that overcomes the issue of low power, thus removing a potential barrier for their use in practice.

Subfields: Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Balanced Linear Contextual Bandits (2018, background): “Methods for reducing the bias due to adaptive data collection have also been studied for non-contextual multi-armed bandits (Villar, Bowden, and Wason 2015), (Nie et al. 2018), but the nature of the estimation in contextual ban-dits is qualitatively different.”
Estimation Considerations in Contextual Bandits (2017, background): “Methods for reducing the bias due to adaptive data collection have also been studied for non-contextual multi-armed bandits [63, 47], but the nature of the estimation in contextual bandits is qualitatively different.”
Simple Bayesian Algorithms for Best Arm Identification (2016, background): “…are extremely expensive, and careful experimentation can help to mitigate the associated costs.1 Multi-armed bandit models of clinical trails date back to Thompson [1933], but bandit algorithms lack statistical power in detecting the best treatment at the end of the trial [Villar et al., 2015].”
Accurate Inference for Adaptive Linear Models (2017, background): “The papers (Villar et al., 2015; Nie et al., 2017) focus on the finite sample effect of the data collection policy on the bias and suggest methods to reduce the bias.”
Multi-Task Learning for Contextual Bandits (2017, methodology): “Contextual bandits have been employed to model various applications like news article recommendation [7], computational advertisement [9], website optimization [20] and clinical trials [19].”
Machine learning applications in drug development (2019, methodology): “The use of ACTs might also be hindered by practical reasons in the clinical trial setting, for instance, when one must deal with considerably delayed feedback, as reported by [77].”
Q-Learning: Theory and Applications (2020, background): “…trade-off in reinforcement learning and other sequential decision settings, and a thorough review is beyond the scope of this article; Auer (2002), Villar et al. (2015), Ghavamzadeh et al. (2015), and Russo et al. (2018) We present one class of solutions to the exploration-exploitation problem…”