SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents

By Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig and 1 more. International Conference on Learning Representations, 2023

368 citations recorded by the source corpus.

Humans are social beings; we pursue social goals in our daily interactions, which is a crucial aspect of social intelligence. Yet, AI systems' abilities in this realm remain elusive. We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and evaluate their social intelligence. In our environment, agents role-play and interact under a wide variety of scenarios; they coordinate, collaborate, exchange, and compete with each other to achieve complex social goals. We simulate the role-play interaction between LLM-based agents and humans within this task space and evaluate their performance with a holistic evaluation framework called SOTOPIA-Eval. With SOTOPIA, we find significant differences between these models in terms of their social intelligence, and we identify a subset of SOTOPIA scenarios, SOTOPIA-hard, that is generally challenging for all models. We find that on this subset, GPT-4 achieves a significantly lower goal completion rate than humans and struggles to exhibit social commonsense reasoning and strategic communication skills. These findings demonstrate SOTOPIA's promise as a general platform for research on evaluating and improving social intelligence in artificial agents.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/f6e893b3e2ee7a62c2fe8a3b0e33920c3e596969

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024, background): “More recently, [Chen et al. , 2023c; Chen et al. , 2023a] introduce frameworks for dynamic multi-agent collaboration, while [Zhou et al. , 2023a; Li et al. , 2023h; Xie et al. , 2023] present platforms and libraries for building autonomous agents, emphasizing their adaptability in task-solving and…”
When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings (2024, unknown): “These studies revealed that GPT-4 struggles to reach human performance with respect to social commonsense reasoning and strategic communication skills ((Zhou et al., 2024)).”