Using cognitive psychology to understand GPT-3

By Marcel Binz, Eric Schulz. 2022

761 citations recorded by the source corpus.

Significance Language models are trained to predict the next word for a given text. Recently, it has been shown that scaling up these models causes them to not only generate language but also to solve challenging reasoning problems. The present article lets a large language model (GPT-3) do experiments from the cognitive psychology literature. We find that GPT-3 can solve many of these tasks reasonably well, despite being only taught to predict future word occurrences on a vast amount of text from the Internet and books. We additionally utilize analysis tools from the cognitive psychology literature to demystify how GPT-3 solves different tasks and use the thereby acquired insights to make recommendations for how to improve future model iterations.

Subfields: Artificial Intelligence, Family Practice, Physiology, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Family Practice

In Physiology

In Large Language Models

Citations in the corpus

ChatGPT outperforms crowd workers for text-annotation tasks (2023, background): “…very well for a wide range of purposes, including ideological scaling (Wu et al., 2023), the classification of legislative proposals (Nay, 2023), the resolution of cognitive psychology tasks, (Binz and Schulz, 2023), and the simulation of human samples for survey research (Argyle et al., 2023).”
Inductive biases for deep learning of higher-level cognition (2020, background): “Our discussion about causality should also suggest that passive observation may be insufficient: in order to capture the causal structure understood by humans, it may be necessary for learning agents to be embedded in an environment in which they can act and thus discover its causal structure [188,1”
Generative Agents: Interactive Simulacra of Human Behavior (2023, methodology): “For example, starting with only a single user-specified notion that one agent wants to throw a Valentine’s Day party, the agents autonomously spread invitations to the party over the next two”
Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, neutral): “Speci ﬁ cally, these works (Binz and Schulz, 2023; Singh et al., 2023) use psychological tests to simulate the human response to test the cognitive ability, emotional intelligence (Elyoseph et al., 2023), and psychological well-being (Li et al., 2022) of LLMs, demonstrating that LLM agents have…”
The potential of generative AI for personalized persuasion at scale (2024, background): “First, LLMs have been shown to bear an uncanny resemblance to humans in how they process information and respond to external stimuli (e.g., 44 ).”
The rise and potential of large language model based agents: a survey (2025, positive): “They conducted psychology experiments about judgment and decision-making to test agent systems [413, 414, 416, 417].”
Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine (2023, background): “68 Meta has recently taken a ﬁ rst exciting step into this direction in form of their CICERO chatbot.”
Whose Opinions Do Language Models Reflect? (2023, background): “One line of work asks whether LMs can replicate results from well-known human experiments, e.g., in cognitive science, social science, and economics (Uchendu et al., 2021; Karra et al., 2022; Aher et al., 2022; Binz & Schulz, 2022; Srivastava et al., 2022).”
Language models show human-like content effects on reasoning (2022, background): “(2020) showed that large language models could perform moderately well on some reasoning tasks, there has been a growing interest in language model reasoning (Binz and Schulz, 2022).”
Testing theory of mind in large language models and humans (2024, background): “In the service of the broader multidisciplinary study of machine behaviour 28 , there have been recent calls for a ‘machine psychology’ 29 that have argued for using tools and paradigms from experimental psychology to systematically investigate the capacities and limits of LLMs 30 .”
Playing repeated games with large language models (2023, neutral): “To avoid contamination through particular choice names or the used framing, we use the neutral options ‘F’ and ‘J’ throughout [Binz and Schulz, 2023].”
PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits (2023, background): “…demonstrate emerging abilities as agents (Andreas, 2022) and exhibit human-like behavior in reasoning (Dasgupta et al., 2022; Webb et al., 2023; Binz and Schulz, 2023; Aher et al., 2023; Wong et al., 2023), role-playing (Wang et al., 2023b; Shao et al., 2023; Wang et al., 2023a), and social…”
The future landscape of large language models in medicine (2023, background): “It is particularly important students learn appropriate prompt engineering, i.e., appropriate and precise phrasing of an appropriate input to achieve the desired output 62 , as misconceived prompts may result in biases or misinformation with potentially serious consequences 4 .”