Transfer of learning by composing solutions of elemental sequential tasks

By Satinder Singh. 2004

220 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Recursively Summarizing Books with Human Feedback (2021, methodology): “Our approach to scalable oversight is directly inspired by Christiano et al. (2018) and Leike et al. (2018), who make use of task decomposition (Singh, 1992; Dayan and Hinton, 1993) and learning from human feedback.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “In this spirit, early work by (Singh, 1992) defined a class of composite tasks called sequential decision tasks, which are expressed as a temporal concatenation of simpler tasks.”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “This makes it hard to evaluate the agent’s ability to compose pre-learned policies to solve previously unseen sequential tasks in a zero-shot fashion unless we re-train the agent on the new tasks in a transfer learning setting (Singh, 1991; 1992; McGovern and Barto, 2002).”
A survey and critique of multiagent deep reinforcement learning (2018, background): “Fedual Networks were in turn, inspired by early work in RL which proposed a hierarchy of Q-learners [82,296]”
Composing Meta-Policies for Autonomous Driving Using Hierarchical Deep Reinforcement Learning (2017, unknown): “studied the case where experience from prior training is transferred between tasks using a Q function and not policies, and others adopted similar transfer models [2, 24].”
The two-dimensional organization of behavior (2011, unknown): “Other modular reinforcement-learning methods [5]–[8], [15]–[23] that break the task down in other ways on the basis of other constraints might also be adaptable to a two-dimensional map, yielding systems with different properties.”
Memorize or generalize? Searching for a compositional RNN in a haystack (2018, background): “As early as [23], a standard approach to neural-network-based composition has been to structure the network into a set of modules”
Multiple Model-Based Reinforcement Learning (2002, methodology): “The modiﬁed CQ-L did not perform signiﬁcantly better than standard, ﬂat Q-learning.”
Value Functions for RL-Based Behavior Transfer: A Comparative Study (2005, methodology): “The “transfer of learning ” approach (Singh 1992) applies specifically to temporally se quential subtasks.”
Multi-Task Reinforcement Learning with Soft Modularization (2020, methodology): “Besides focusing on the gradients, another direction is to design compositional model with multiple modules (Singh, 1992; Devin et al., 2017; Andreas et al., 2017; Rusu et al., 2016; Qureshi et al., 2020; Peng et al., 2019; Haarnoja et al., 2018a; Sahni et al., 2017).”
Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning (2018, methodology): “…work on giving an agent knowledge about the reward function focus on defining a task specification language, usually based on sub-goal sequences (Singh, 1992a;b) or linear temporal logic (Li et al., 2017; Camacho et al., 2017; Littman et al., 2017; Toro Icarte et al., 2018; Hasanbeig et al.,…”
Exploration from Generalization Mediated by Multiple Controllers (2013, background): “Consider, for instance the famous taxicab problem, involving navigation in a maze between many, or indeed all, points (e.g., Dietterich 1998; Singh 1992; Thrun and Schwartz 1995).”
CHILD: A First Step Towards Continual Learning (1997, methodology): “The importance of hierarchy in adaptive systems that perform temporal tasks has been noted often, and many systems have been proposed in which hierarchical architectures are developed by hand top down as an efficient method for modularizing large temporal tasks (Albus, 1979; Dayan and Hinton, 1993; ”
Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning (2020, methodology): “t al., 2020), and reward shaping (Ng et al., 1999). Hierarchical reinforcement learning is the most successful methodology to exploit decompositions in RL. Some foundational HRL works include H-DYNA (Singh, 1992a), MAXQ (Dietterich, 2000), HAMs (Parr & Russell, 1998), and Options (Sutton et al.,”
Robust non-linear control through neuroevolution (2003, background): “A number of researchers have used shaping to make learning complex tasks tractable (Colombetti and Dorigo 1992; Jacobs et al. 1991; Perkins and Hayes 1996; Singh 1992).”
Multitask reinforcement learning on the distribution of MDPs (2003, methodology): “[8, 10] Among them, a modular-approach such as Singh’s CQ-learning[5] is one promising way, reusing past learning experiences by switching value function modules obtained before.”
From Baby Steps to Leapfrog: How “Less is More” in Unsupervised Dependency Parsing (2010, background): “Although not representative of the full grammar, short sentences capture enough information to paint most of the picture needed by slightly longer sentences.”