Instructable Intelligent Personal Agent

By Amos Azaria, Jayant Krishnamurthy, Tom Michael Mitchell. 2016

85 citations recorded by the source corpus.

Unlike traditional machine learning methods, humans often learn from natural language instruction. As users become increasingly accustomed to interacting with mobile devices using speech, their interest in instructing these devices in natural language is likely to grow. We introduce our Learning by Instruction Agent (LIA), an intelligent personal agent that users can teach to perform new action sequences to achieve new commands, using solely natural language interaction. LIA uses a CCG semantic parser to ground the semantics of each command in terms of primitive executable procedures defining sensors and effectors of the agent. Given a natural language command that LIA does not understand, it prompts the user to explain how to achieve the command through a sequence of steps, also specified in natural language. A novel lexicon induction algorithm enables LIA to generalize across taught commands, e.g., having been taught how to "forward an email to Alice," LIA can correctly interpret the command "forward this email to Bob." A user study involving email tasks demonstrates that users voluntarily teach LIA new commands, and that these taught commands significantly reduce task completion time. These results demonstrate the potential of natural language instruction as a significant, under-explored paradigm for machine learning.

Subfields: Artificial Intelligence, LLM Agents & Tool Use, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In LLM Agents & Tool Use

Citations in the corpus

Joint Concept Learning and Semantic Parsing from Natural Language Explanations (2017, background): “The general idea of learning through human interactions has previously been explored in settings such as behavioral programming (Harel et al., 2012), natural language programming (Biermann, 1983), learning by instruction (Azaria et al., 2016), etc.”
Naturalizing a Programming Language via Interactive Learning (2017, background): “Azaria et al. (2016) presents Learning by In-struction Agent (LIA), which also advocates learning from users.”
Conversational Neuro-Symbolic Commonsense Reasoning (2020, background): “The advent of intelligent agents and advancements in natural language processing have given learning from conversational interactions a good momentum in the last few years [15–23].”
Learning Out-of-Vocabulary Words in Intelligent Personal Agents (2018, background): “The problem of learning new command words have also been studied in CCG based parsers [Thomason et al. , 2015; Azaria et al. , 2016].”
Conversational Multi-Hop Reasoning with Neural Commonsense Knowledge and Symbolic Logic Rules (2021, background): “On the other hand, instructable agents (Azaria et al., 2016; Labutov et al., 2018; Li et al., 2018, 2017b,a; Guo et al., 2018; Mohan and Laird, 2014; Mininger and Laird, 2018; Mohan et al., 2012), can be taught new tasks through natural language instructions/demonstrations.”