Human few-shot learning of compositional instructions

By Brenden M. Lake, Tal Linzen, Marco Baroni. 2019

119 citations recorded by the source corpus.

People learn in fast and flexible ways that have not been emulated by machines. Once a person learns a new verb "dax," he or she can effortlessly understand how to "dax twice," "walk and dax," or "dax vigorously." There have been striking recent improvements in machine learning for natural language processing, yet the best algorithms require vast amounts of experience and struggle to generalize new concepts in compositional ways. To better understand these distinctively human abilities, we study the compositional skills of people through language-like instruction learning tasks. Our results show that people can learn and use novel functional concepts from very few examples (few-shot learning), successfully applying familiar functions to novel inputs. People can also compose concepts in complex ways that go beyond the provided demonstrations. Two additional experiments examined the assumptions and inductive biases that people make when solving these tasks, revealing three biases: mutual exclusivity, one-to-one mappings, and iconic concatenation. We discuss the implications for cognitive modeling and the potential for building machines with more human-like language learning capabilities.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Prompting & Reasoning, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Prompting & Reasoning

In Cognitive Science & Psychology

Citations in the corpus

Environmental drivers of systematicity and generalization in a situated agent (2019, background): “Systematic generalisation (Plaut, 1999; Bahdanau et al., 2018; Lake et al., 2019) is a desirable characteristic for a computational model because it suggests that if the model can understand components (or words) in certain combinations, it should also understand the same components in different…”
Good-Enough Compositional Data Augmentation (2019, background): “Human learners make judgments like (2) about novel lexical items (Berko, 1958) and fragments of novel languages (Lake et al., 2019).”
One-Shot Learning from a Demonstration with Hierarchical Latent Language (2022, unknown): “Language-Based Generalization in Humans and Models Lake and Baroni (2018) show that RNN-based sequence models struggle to perform systematic compositional generalization based on abstract linguistic rules, while humans are extremely effective at it given few examples (Lake et al., 2019).”
Multimodal Graph Networks for Compositional Generalization in Visual Question Answering (2020, background): “[35] proposed a task of translating natural language instructions into explicit action sequences, with the goal of composing terms like ‘jump‘ and ‘twice‘ to intrepret compound instructions.”
Symbolic Behaviour in Artificial Intelligence (2021, background): “We acknowledge the graded, flawed nature of human behaviour—even adult humans are often far from perfectly systematic, achieving compositional generalization performance of only 70-90% even within relatively simple domains [108].”
Compositional Generalization for Primitive Substitutions (2019, methodology): “The dataset (Lake et al., 2019)”
Few-Shot Learning With a Strong Teacher (2021, background): “T HE intriguing ability that humans can learn new knowledge with few examples and then generalize it to novel environments [1], [2] has led to a community-wide enthusiasm in artiﬁcial intelligence towards few-shot learning (FSL) 1 , especially in the applications of visual recognition [3], [4],…”
Compositional generalization through meta sequence-to-sequence learning (2019, methodology): “To answer the queries, a model must acquire two abilities inspired by human generalization patterns [18]: 1) using isolated symbol mappings to translate concatenated symbol sequences, and 2) using mutual exclusivity (ME) to resolve unseen mappings.”
Compositional Generalization by Learning Analytical Expressions (2020, methodology): “Besides them, we also conduct experiments on the Simple task of SCAN which requires no compositional generalization capacity, and the Limit task of MiniSCAN [21] which evaluates if models can learn compositional generalization when given limited (i.e. 14 ) training data.”
Learning to Recombine and Resample Data for Compositional Generalization (2020, methodology): “For example, learners should generalize from actions seen in isolation to more complex commands involving those actions (Lake et al., 2019), and from relations of the form r(a,b) to r(b,a) (Keysers et al., 2020; Bahdanau et al., 2019b).”
Emergent Systematic Generalization in a Situated Agent (2019, background): “Systematic generalization (Plaut, 1999; Bahdanau et al., 2018; Lake et al., 2019) is a desirable characteristic for a computational model because it suggests that if the model can understand components (or words) in certain combinations, it should also understand the same components in different com”
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2020, background): “First, humans have been shown to be adept compositional learners (Lake et al., 2019).”
Learning Compositional Rules via Neural Program Synthesis (2020, methodology): “In Lake et al. (2019), human subjects were allowed to study the 14 example ‘support instructions’ in Figure 2, which demonstrate how to transform a sequence of nonce words into a sequence of colored circles.”
Compositional Generalization via Neural-Symbolic Stack Machines (2020, background): “Given a set of basic components and a few demonstrations of their combinations, a person could effectively capture the underlying compositional rules, and generalize the knowledge to novel combinations [9, 34, 27, 26].”
A Benchmark for Systematic Generalization in Grounded Language Understanding (2020, background): “Crucially, the difference between these works and ours is that we rely on evidence that humans systematically generalize in language [9, 33, 28] and only test for linguistic, rule-based generalization.”
Teaching Machine Comprehension with Compositional Explanations (2020, background): “One strength of human cognition is the ability to generalize from relatively few examples; shown only a few instances of a problem and solution, humans often deduce patterns more readily than a machine, typically by bringing to bear a wealth of background information about what “really matters” in e”
Leveraging Language to Learn Program Abstractions and Search Heuristics (2021, background): “Future human experiments could explore LAPS as a cognitive model, combining paradigms for studying language learning with those for studying non-linguistic abstraction and search (e.g. Smith et al. 2003; Hawkins et al. 2019; Lake et al. 2015; 2019; Tian et al. 2020).”