Accelerating Reinforcement Learning by Composing Solutions of Automatically Identified Subtasks

By C. Drummond. 2011

108 citations recorded by the source corpus.

This paper discusses a system that accelerates reinforcement learning by using transfer from related tasks. Without such transfer, even if two tasks are very similar at some abstract level, an extensive re-learning effort is required. The system achieves much of its power by transferring parts of previously learned solutions rather than a single complete solution. The system exploits strong features in the multi-dimensional function produced by reinforcement learning in solving a particular task. These features are stable and easy to recognize early in the learning process. They generate a partitioning of the state space and thus the function. The partition is represented as a graph. This is used to index and compose functions stored in a case base to form a close approximation to the solution of the new task. Experiments demonstrate that function composition often produces more than an order of magnitude increase in learning rate compared to a basic reinforcement learning algorithm.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Learning modular neural network policies for multi-task and multi-robot transfer (2016, methodology): “Another transfer approach used by [21] is to split each task into sub-tasks and transfer the sub-tasks between tasks.”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, background): “Another class of algorithms (Drummond, 2002; Fox et al., 2017; Şimşek et al., 2005) focus solely on automatically discovering these skills or subgoals.”
Behavior transfer for value-function-based reinforcement learning (2005, background): “the subtasks can be automatically identified [7] and leveraged to increase learning rates.”
Transfer learning in multi-agent systems through parallel transfer (2013, background): “There have been many attempts to accelerate learning in RL (Powell & Ma, 2011; Drummond, 2011).”
AUTOMATED DISCOVERY OF OPTIONS IN REINFORCEMENT LEARNING (2003, background): “Drummond’s algorithm focuses on speeding up learning by reusing the value functions of previously learned tasks [9][10].”
Transfer Learning via Inter-Task Mappings for Temporal Difference Learning (2007, background): “In some problems where subtasks are clearly defined by state features, the subtasks can be automatically identified (Drummond, 2002) and leveraged to increase learning rates.”
Value Functions for RL-Based Behavior Transfer: A Comparative Study (2005, background): “In some problems where subtasks are clearly defined by features, the subtasks can be automatically identified (Drummond 2002) and leveraged to increase learning rates.”
Transfer Learning for Reinforcement Learning Domains: A Survey (2009, background): “In contrast, our definition of transfer learning focuses on agents successfully reusing internalknowledge on novel problems.”
Transfer in Reinforcement Learning: A Framework and a Survey (2012, methodology): “Drummond (2002) propose a similar method which identifies subtasks and features according to the analysis of the dynamics of the source tasks.”
Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes (2007, result): “While the overall goal of learning representations is not new within the context of MDPs— it has been addressed by Dayan (1993) and Drummond (2002) among others—our approach is substantially different from previous work.”