Sequential Transfer in Multi-armed Bandit with Finite Set of Models

By Mohammad Gheshlaghi Azar, Alessandro Lazaric, Emma Brunskill. 2013

118 citations recorded by the source corpus.

Learning from prior tasks and transferring that experience to improve future performance is critical for building lifelong learning agents. Although results in supervised and reinforcement learning show that transfer may significantly improve the learning performance, most of the literature on transfer is focused on batch learning tasks. In this paper we study the problem of sequential transfer in online learning, notably in the multi-armed bandit framework, where the objective is to minimize the total regret over a sequence of tasks by transferring knowledge from prior tasks. We introduce a novel bandit algorithm based on a method-of-moments approach for estimating the possible tasks and derive regret bounds for it.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Reinforcement Learning in Rich-Observation MDPs using Spectral Methods (2016, background): “For multi-armed bandits, Gheshlaghi-Azar et al. [10] and Maillard and Mannor [19] assume that a bandit problem is generated from an unknown (latent) ﬁnite set and show how the regret can be signiﬁcantly reduced by learning this set.”
Transferable Contextual Bandit for Cross-Domain Recommendation (2018, background): “Though tUCB (Azar, Lazaric, and Brunskill 2013) and B-kl-UCB (Zhang and Bareinboim 2017) consider transfer learning for the bandit problem by transferring the estimated upper confidence bound, they differ from ours greatly. tUCB considers context-free multi-armed bandit problems which are…”
Reinforcement Learning of POMDPs using Spectral Methods (2016, methodology): “Beside its practical success in offline problems, policy search has been successfully integrated with efficient exploration–exploitation techniques and shown to achieve small regret (Gheshlaghi-Azar et al., 2013, 2014).”
Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret (2015, background): “Mohammad Gheshlaghi Azar, Alessandro Lazaric, & Emma Brunskill.”
Bounded Regret for Finite-Armed Structured Bandits (2014, methodology): “While the setting, objective and analysis are different, the algorithm is essentially the same as “model UCB” introduced in [8].”
Online Clustering of Bandits (2014, background): “In (Azar et al., 2013), the authors deﬁne a transfer learning problem within a stochastic multiarmed bandit setting, where a prior distribution is deﬁned over the set of possible models over the tasks.”