Building Portable Options: Skill Transfer in Reinforcement Learning

By George Dimitri Konidaris, Andrew Barto. 2007

300 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Learning Parameterized Skills (2012, methodology): “Konidaris and Barto (2007) learn reusable options by representing them in an agent-centered state space but do not address the problem of how to construct skills for solving a family of related tasks.”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, methodology): “…delayed rewards or other long-term structure are often dif-ﬁcult to solve with ﬂat, monolithic policies, and a long line of prior work has studied methods for learning hierarchical policy representations (Sutton et al., 1999; Diet-terich, 2000; Konidaris & Barto, 2007; Hauser et al., 2008).”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “Typically these have the form of a meta controller and a set of lower-level controllers for subtasks (Sutton et al., 1999; Dietterich, 2000; Parr and Russell, 1997; Ghavamzadeh and Mahadevan, 2003; Konidaris et al., 2012; Konidaris and Barto, 2007).”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “These approaches rely on hand-crafted rewards based on prior domain knowledge [21, 18, 22, 40] or diversity-encouraging rewards like mutual information [6, 10].”
Stochastic Neural Networks for Hierarchical Reinforcement Learning (2016, background): “This speciﬁc structural assumption has been studied in the past as sharing the same agent-space (Konidaris & Barto, 2007).”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “These rewards include mutual information-based diversity rewards [12, 17], hand-crafted rewards based on domain knowledge [29, 24, 30, 59], and goal-oriented rewards [15, 49, 3, 62, 36, 37].”
Robot learning from demonstration by constructing skill trees (2012, background): “In addition, skills represented using agent-centric features (such as in our uBot example) can be detached from a problem-specific setting and transferred to new problems (Konidaris and Barto 2007).”
Transfer of samples in batch reinforcement learning (2008, background): “In ( Konidaris & Barto, 2007 ), a set of options is learned in an agent space defined by a set of features shared across the tasks, thus making the options reusable even in tasks with different state spaces.”
SWIRL: A sequential windowed inverse reinforcement learning algorithm for robot tasks with delayed rewards (2018, background): “…(Brooks 1986) and by learning them given various manual specifications: state abstractions (Dayan and Hinton 1992; Hengst 2002; Kolter et al. 2007b; Konidaris and Barto 2007), a set of waypoints (Kaelbling 1993), lowlevel skills (Huber and Grupen 1997; Bacon and Precup 2015), a set of finite-state…”
Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining (2009, background): “If the agent faces a sequence of tasks having the same state space, then options learned in it are portable [14, 15, 16, 17]; if it faces a sequence of tasks having different but related state spaces, then the options must be learned using features common to all the tasks [22].”
PAC-inspired Option Discovery in Lifelong Reinforcement Learning (2014, background): “Homomorphisms (Soni & Singh, 2006) and shared features (Konidaris & Barto, 2007) can be used to discover useful options across multiple tasks with different state/action spaces.”
Measuring the Distance Between Finite Markov Decision Processes (2016, unknown): “instances [19], Q-values [29] and policies [16]), and use the knowledge to improve the performance of solving the new MDP.”
Autonomous Skill Acquisition on a Mobile Manipulator (2011, background): “Additionally, a change in abstraction is a useful measure of subtask boundaries, and the use of agent-centric abstractions facilitates skill transfer (Konidaris and Barto 2007).”