Theory of Mind May Have Spontaneously Emerged in Large Language Models

By Michal Kosinski. 2023

268 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Large Language Models, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Denny Zhou
2. Xinyun Chen
3. Karthik Narasimhan
4. Shunyu Yao
5. Tengyu Ma
6. Percy Liang
7. Yang Liu
8. Xuezhi Wang

In Control and Systems Engineering

1. Karthik Narasimhan
2. Tengyu Ma
3. Tomer D. Ullman
4. Yang Liu
5. Denny Zhou

In Large Language Models

1. Yang Liu
2. Denny Zhou
3. Tengyu Ma
4. Xinyun Chen
5. Shunyu Yao
6. Karthik Narasimhan
7. Peng Li
8. Xuezhi Wang

In Cognitive Science & Psychology

1. RosaI. Arriaga
2. Karthik Narasimhan
3. Adam Tauman Kalai
4. Tomer D. Ullman
5. Timm Teubner
6. Denny Zhou

Citations in the corpus

• Welcome to the Era of ChatGPT et al. (2023, background): “, testing the ability to impute unobservable mental states to others) yields correct-response rates of 93% (comparable to nine-year old children), while ‘‘models published before 2022 show virtually no ability’’ to solve such ToM tasks (Kosinski 2023).”
• Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies (2022, background): “Concurrent work by Ullman (2023) shows that although GPT-3 previously showed success on Theory of Mind psychology tasks (Kosinski, 2023), it fails on prompts with several types of directed variations, illustrating the necessity of evaluating the robustness of observed effects using alternative…”
• GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (2023, methodology): “called GPT-1 that was created using unlabeled data and then given to users to fine-tune to carry out subsequent tasks like sentiment analysis, categorization, and question-answering [22].”
• Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf (2023, background): “Furthermore, studies have shown that LLMs exhibit a certain degree of theory of mind capabilities (Bubeck et al., 2023; Shapira et al., 2023; Kosinski, 2023), as well as the potential to simulate believable human behaviors (Park et al.”
• Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks (2023, background): “Kosinski found that for the content prompt, [P(popcorn) = 100%; P(chocolate) = 0%].”
• Reflexion: language agents with verbal reinforcement learning (2023, background): “Kosinski [8] show notable performance on theory of mind (ToM) tasks at the level of young to middle-aged human children.”
• Large Language Models as Tool Makers (2023, background): “…across a broad array of NLP tasks (Brown et al., 2020; Chowdhery et al., 2022; Zhang et al., 2022; Hoffmann et al., 2022; OpenAI, 2023; Google, 2023) and have even shown promising signs of achieving certain aspects of artificial general intelligence (Bubeck et al., 2023; Kosinski, 2023).”
• The potential of generative AI for personalized persuasion at scale (2024, background): “For example, recent work suggests that a central psychological process in personalized persuasion, theory of mind (e.g., representing other people’s mental states), may have spontaneously emerged in LLMs (e.g., 45,46 ).”
• A Survey of Large Language Models (2023, unknown): “5 series achieves comparable performance with nine-year-old children in ToM task [418].”