Placing language in an integrated understanding system: Next steps toward human-level performance in neural language models

By James L. McClelland, Felix Hill, Maja R. Rudolph, Jason Baldridge, Hinrich Schütze. 2020

96 citations recorded by the source corpus.

Language is crucial for human intelligence, but what exactly is its role? We take language to be a part of a system for understanding and communicating about situations. In humans, these abilities emerge gradually from experience and depend on domain-general principles of biological neural networks: connection-based learning, distributed representation, and context-sensitive, mutual constraint satisfaction-based processing. Current artificial language processing systems rely on the same domain general principles, embodied in artificial neural networks. Indeed, recent progress in this field depends on query-based attention, which extends the ability of these systems to exploit context and has contributed to remarkable breakthroughs. Nevertheless, most current models focus exclusively on language-internal tasks, limiting their ability to perform tasks that depend on understanding situations. These systems also lack memory for the contents of prior situations outside of a fixed contextual span. We describe the organization of the brain’s distributed understanding system, which includes a fast learning system that addresses the memory problem. We sketch a framework for future models of understanding drawing equally on cognitive neuroscience and artificial intelligence and exploiting query-based attention. We highlight relevant current directions and consider further developments needed to fully capture human-level language understanding in a computational system.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Large Language Models

In Natural Language Processing

Citations in the corpus

Can language models learn from explanations in context? (2022, background): “Differences between human and model responses may originate in part from the impoverished experience of LMs, which do not experience the broader context or situations to which language refers (McClelland et al., 2020).”
Shared computational principles for language processing in humans and deep language models (2022, background): “Furthermore, DLMs avoid the circularity built into many psycholinguistic language models that rely on linguistic terms to explain how language is encoded in neural substrates 19,62 .”
Mind's Eye: Grounded Language Model Reasoning through Simulation (2022, background): “However, these models do not experience firsthand the situations that are described by the language (McClelland et al., 2020), and lack the ability to find the correct answers by performing experiments like humans.”
Data Distributional Properties Drive Emergent In-Context Learning in Transformers (2022, result): “Our results may also relate to complementary learning systems theory (Kumaran et al., 2016; Mc-Clelland and O’Reilly, 1995) and its application to language understanding in the brain (McClelland et al., 2020).”
Symbolic Behaviour in Artificial Intelligence (2021, background): “[44] James L McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze.”
Language models show human-like content effects on reasoning (2022, background): “More fundamentally, language models do not directly experience the situations to which language refers (McClelland et al., 2020); grounded experience (for instance the capacity to simulate the physical turning of cards on a table) presumably underpins some human beliefs and reasoning.”