Emergent analogical reasoning in large language models

By Taylor W. Webb, Keith J. Holyoak, Hongjing Lu. 2022

497 citations recorded by the source corpus.

The recent advent of large language models has reinvigorated debate over whether human cognitive capacities might emerge in such generic models given sufficient training data. Of particular interest is the ability of these models to reason about novel problems zero-shot, without any direct training. In human cognition, this capacity is closely tied to an ability to reason by analogy. Here we performed a direct comparison between human reasoners and a large language model (the text-davinci-003 variant of Generative Pre-trained Transformer (GPT)-3) on a range of analogical tasks, including a non-visual matrix reasoning task based on the rule structure of Raven’s Standard Progressive Matrices. We found that GPT-3 displayed a surprisingly strong capacity for abstract pattern induction, matching or even surpassing human capabilities in most settings; preliminary tests of GPT-4 indicated even better performance. Our results indicate that large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems. Webb et al. show that new artificial intelligence language models, such as Generative Pre-trained Transformer 3, are able to solve analogical reasoning problems at a human-like level of performance.

Subfields: Artificial Intelligence, Cultural Studies

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

Citations in the corpus

BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “Our hypothesis is that as BASE TTS increases in model capacity and trains over more data, the model will start to acquire these abilities, following evidence that scaling in these dimensions begets qualitative ability jumps [1, 32, 33].”
Large Language Models as Analogical Reasoners (2023, background): “A recent work also evaluates the ability of language models to identify analogies (Webb et al., 2023; Hu et al., 2023).”
Testing theory of mind in large language models and humans (2024, unknown): “A systematic experimental approach to studying theory of mind in LLMs involves using a diverse set of theory of mind measures, delivering multiple repetitions of each test, and having clearly defined benchmarks of human performance against which to compare 31 .”
Playing repeated games with large language models (2023, unknown): “They can, for example, solve analogical reasoning tasks [Webb et al., 2022], program web applications [Chen et al.”
A Comprehensive Overview of Large Language Models (2023, unknown): “1: The trends in the number of LLM models introduced over the years.”
Position: Levels of AGI for Operationalizing Progress on the Path to AGI (2023, unknown): “…tasks (per Principle 3), measuring diverse properties including (but not limited to) linguistic intelligence, mathematical and logical reasoning (Webb et al., 2023), spatial reasoning, interpersonal and intra-personal social intelligences, the ability to learn new skills (Chollet, 2019), and…”
Reasoning with Language Model Prompting: A Survey (2022, background): “We argue that the generalizable reasoning may be closely related to analogy reasoning (Chen et al., 2022a; Webb et al., 2022), causal reasoning (Feder et al.”
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, background): “Other works focus on more specific abilities such as mathematical skills (Davis, 2023b), reasoning (Webb et al., 2022a; Qiao et al., 2022).”