Autonomous transfer for reinforcement learning

By Matthew E. Taylor, Gregory Kuhlmann, Peter Stone. 2008

134 citations recorded by the source corpus.

Recent work in transfer learning has succeeded in making reinforcement learning algorithms more efficient by incorporating knowledge from previous tasks. However, such methods typically must be provided either a full model of the tasks or an explicit relation mapping one task into the other. An autonomous agent may not have access to such high-level information, but would be able to analyze its experience to find similarities between tasks. In this paper we introduce Modeling Approximate State Transitions by Exploiting Regression (MASTER), a method for automatically learning a mapping from one task to another through an agent's experience. We empirically demonstrate that such learned relationships can significantly improve the speed of a reinforcement learning algorithm in a series of Mountain Car tasks. Additionally, we demonstrate that our method may also assist with the difficult problem of task selection for transfer.

Subfields: Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Learning Parameterized Skills (2012, background): “[75] transfer Q-values between tasks defined over different state and action spaces.”
Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, background): “inapov, Taylor, and Stone (a) (b) Figure 2: Performance metrics for transfer learning using (a) weak transfer and (b) strong transfer with oset curves. be specied by hand, or learned automatically (Taylor et al., 2008; Ammar et al., 2015). Other methods assume the transition or reward functions do n”
Unsupervised Cross-Domain Transfer in Policy Gradient Reinforcement Learning via Manifold Alignment (2015, background): “…WILL APPEAR IN THE AAAI DIGITAL LIBRARY tic knowledge about state features between two tasks (Liu & Stone 2006), background knowledge about the range or type of state variables (Taylor et al. 2007b), or transition models for each possible mapping could be generated and tested (Taylor et al. 2008).”
RLLib: C++ Library to Predict, Control, and Represent Learnable Knowledge Using On/Off Policy Reinforcement Learning (2015, methodology): “MC3 Gq describes the three dimensional mountain car [30] solved using greedyGQ algorithm [14].”
Source Task Creation for Curriculum Learning (2016, background): “For example, 3v2 keepaway serving as a source for 4v3 keepaway [26], 2D mountain car as a source for 3D mountain car [25], or varying parameters of physical systems [2].”
Transfer in Reinforcement Learning: A Framework and a Survey (2012, background): “For instance, in (Taylor et al, 2008b) target samples are used to identify the best mapping between source and target state-action variables and thus to transform the source policy into a target policy used to initialize the learning process in the target task.”
Transfer Learning by Mapping with Minimal Target Data (2008, methodology): “In the area of reinforcement learning, Taylor et al. (2008) present a method for mapping the feature spaces across tasks by using samples from runs in the two tasks.”
FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented Microservices (2020, background): “Related work on base model selection and task similarity can be found in [105, 106], but the base model that FIRM uses for transfer learning is always the RL model learned in the general case because it has been shown in evaluation to be comparable with specialized models.”
Policy Transfer using Reward Shaping (2015, background): “The 3D version of the task [18] (or 4D as some would have it) is very similar, except that the terrain is three dimensional, and the car can apply force along two axes, although only one at a time (A = {East,West,North, South,Neutral}).”
Reinforcement Learning Transfer Using a Sparse Coded Inter-task Mapping (2011, methodology): “Unlike all other existing methods (to the best of our knowledge) and complementary to our previous work [1, 14, 15], we assume differences among all the variables of Markov Decision Processes describing the source and target tasks, and focus on learning”
Transferring Instances for Model-Based Reinforcement Learning (2008, methodology): “In this work we assume that the inter-task mapping in Table 1 is provided to the agent, but ot her work [19] has demonstrated that the same mapping may be learn ed Inter-task Mapping for Mountain Car”
Automatically Mapped Transfer between Reinforcement Learning Tasks via Three-Way Restricted Boltzmann Machines (2013, background): “For example, semantic knowledge about state features between two tasks may be used [8,10], background knowledge about the range or type of state variables can be used [14,19], or transition models for each possible mapping could be generated and tested [16].”