Using Confidence Bounds for Exploitation-Exploration Trade-offs

By Peter Auer. 2003

2,118 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “Sample efficient exploration-exploitation trade-off is extensively studied in RL literature from Bandits [6], MDPs [26, 14, 5, 10, 23], and Monte Carlo sampling [25, 27] to partially observable and partial monitoring [9, 11] in low dimensional environment, leaves it open in high dimensional RL [36, ”
A Survey on Neural Architecture Search (2019, methodology): “A policy based on upper confidence bound (Auer, 2002) is utilized to explore this tree. The work of Negrinho and Gordon (2017) adopts the vanilla UCT algorithm with bisection for neural architecture search. Wistuba (2018b) builds on this work and employs function-preserving operations to speed up th”
Survey on Automated Machine Learning (2019, background): “In the literature many different policies have been proposed, for example greedy and -greedy methods (Langford and Zhang, 2008) or the upper confidence bound (Auer, 2002).”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, methodology): “Originally, the upper confidence was given by frequentist Chernoff–Hoeffding bounds [8].”
Balanced Linear Contextual Bandits (2018, neutral): “(For some results, e.g., (Auer 2002), α needs to be slowly increasing in t .)”
Meta-Learning with Temporal Convolutions (2017, background): “There also exist several hand-designed algorithms [7, 3, 2] with optimality guarantees that can serve as baselines.”
Off-policy evaluation for slate recommendation (2016, background): “, linear) models relating the metrics to the features describing a slate [2, 31, 15, 10, 29]; this can lead to bias if the model is inaccurate (e.”
Multi-agent reinforcement learning as a rehearsal for decentralized planning (2016, positive): “In our experiments, we use an upperconfidence bound (UCB) [3] approach for SELECTACTIONMDP; however, other action selection approaches might be applied successfully too.”
The offset tree for learning with partial labels (2008, background): “The interactive version has been analyzed under various additional assumptions [4, 10, 13, 14, 16, 18], including payoffs as a linear function of the side information [1, 2].”
Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems (2012, neutral): “2 was introduced by Auer et al. [2002a] for bounded random variables.”
Multi-fidelity Bayesian Optimisation with Continuous Approximations (2017, methodology): “GP-UCB models f as a GP and uses upper conﬁdence bound (UCB) (Auer, 2003) techniques to determine the next point for evaluation.”
Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits (2014, methodology): “LinUCB (Auer, 2002; Chu et al., 2011) has been quite effective in past evaluations (Li et al.”
Fair Algorithms for Infinite and Contextual Bandits (2016, background): “…design fair algorithms when at most some ﬁnite number k of individuals arrives in any round (the linear contextual bandits problem [Abe et al., 2003, Auer, 2002]), as well as when m individuals may be chosen in each round (the “multiple play" introduced and studied absent fairness in Anantharam et…”
Parallelizing Exploration-Exploitation Tradeoffs with Gaussian Process Bandit Optimization (2012, background): “(2010), guided by the success of upper-confidence-based sampling approaches for multi-armed bandit problems (Auer, 2002), analyzed the Gaussian process Upper Confidence Bound (GP-UCB) decision rule,”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, background): “Despite compelling theoretical results, these methods are mainly model based and limited to tabular settings [33, 11].”
Gaussian Processes for Data-Efficient Learning in Robotics and Control (2015, neutral): “It is, however, also possible to explicitly encourage additional exploration in a UCB (upper confidence bounds) sense [6]: Instead of summing up expected immediate costs, see (2), we would add the sum of cost standard deviations, weighted by a factor k 2 IR.”