Algorithmic and Human Teaching of Sequential Decision Tasks

By Maya Cakmak, Manuel Lopes. 2012

117 citations recorded by the source corpus.

A helpful teacher can significantly improve the learning rate of a learning agent. Teaching algorithms have been formally studied within the field of Algorithmic Teaching. These give important insights into how a teacher can select the most informative examples while teachinga new concept. However the field has so far focused purely on classification tasks. In this paper we introducea novel method for optimally teaching sequential decision tasks. We present an algorithm that automatically selects the set of most informative demonstrations andevaluate it on several navigation tasks. Next, we explore the idea of using this algorithm to produce instructions for humans on how to choose examples when teaching sequential decision tasks. We present a user study that demonstrates the utility of such instructions.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Machine Teaching: An Inverse Problem to Machine Learning and an Approach Toward Optimal Education (2015, background): “…in machine teaching, such as the sequential teaching setting (Bengio et al. 2009; Khan, Zhu, and Mutlu 2011; McCandliss et al. 2002; Kumar, Packer, and Koller 2010; Lee and Grauman 2011; Cakmak and Lopes 2012; Pashler and Mozer 2013; Kobayashi and Shinohara 2009; Balbach and Zeugmann 2009).”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “Other paradigms for human-in-the-loop policy learning include collaboration [14, 15], teaching the robot through informative sample selection [7, 9, 17], and leveraging physical kinesthetic corrections [5, 6].”
Cooperative Inverse Reinforcement Learning (2016, background): “Cakmak & Lopes (2012) consider an application of optimal teaching where the goal is to teach the learner the reward function for an MDP .”
Machine Teaching for Bayesian Learners in the Exponential Family (2013, background): “Interactive systems have been built which employ or study teaching heuristics [4, 6].”
Teacher Improves Learning by Selecting a Training Subset (2018, background): “Machine teaching applications include education [14, 33, 39, 27, 13, 34], computer security [2, 1, 32], and interactive machine learning [40, 12, 24].”
Multi-Armed Bandits for Intelligent Tutoring Systems (2013, background): “Work within AT tries to identify the teachability of different concept classes and devise efficient algorithms that produce optimal teaching sequences [5, 29, 15].”
Exploration from Demonstration for Interactive Reinforcement Learning (2016, background): “Several papers have shown how the policy information in human demonstrations can be used for inverse optimal control [28, 2], for teaching [8], to seed an agent’s exploration [5, 38], and in some cases be used entirely in place of exploration [21, 33].”
Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning (2015, background): “One approach to learning from demonstration is the use of algorithms for inverse reinforcement learning [1,5], where the learner attempts to identify the reward function of a Markov decision process that is consistent with a user’s demonstrated actions, and identify a full policy that is optimal und”
Challenges in Shared-Environment Human-Robot Collaboration (2013, background): “Work by Cakmak and Lopes details an algorithm capable of identifying the most desirable areas of training data through analysis within sequential decision problems [10].”
Becoming the expert - interactive multi-class machine teaching (2015, background): “However, it is worth noting that different types of teaching tasks have been explored in the literature, e.g . sequential decision tasks [8].”
Policy Shaping with Human Teachers (2015, background): “For example, if human teachers see what types of demonstrations will be useful, they produce more useful demonstrations [Cakmak and Lopes, 2012].”
An Overview of Machine Teaching (2018, unknown): “[8] for initial results in this direction.”
Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications (2018, methodology): “Our ﬁrst application is to provide a lower bound on the sample complexity of learning a reward function via active queries (Lopes, Melo, and Montesano 2009; Cui and Niekum 2017; Brown, Cui, and Niekum 2018).”
Optimal Teaching for Limited-Capacity Human Learners (2014, background): “Such a result would establish a conceptual link between idealization manipulations from psychology and optimal teaching procedures from machine learning [7, 6, 8, 2, 9, 10, 11].”
Online Transfer Learning in Reinforcement Learning Domains (2015, methodology): “Cakmak and Lopes (2012) developed a teaching method which is based on algorithm teaching, but their work focuses on one-time optimal teaching sequence computing, which lacks the online setting.”
Learning something from nothing: Leveraging implicit human feedback strategies (2014, background): “There is also a growing body of work that examines how humans want to teach agents by providing demonstrations in a sequential decision task [4], or by selecting a sequence of data in a classification task [7].”
Showing versus doing: Teaching by demonstration (2016, background): “Although recent work has examined user-assisted teaching [8], identiﬁed legible motor behavior in human-machine coordination [9], and analyzed reward coordination in game theoretic terms [11], previous work has yet to successfully model how people naturally teach reward functions by demonstration.”
The strategic student approach for life-long exploration and learning (2012, background): “A first family of approaches have considered mechanisms where ordered training examples are provided by an external teacher, such as in optimal teaching techniques [1] or in curriculum learning [2], [3].”