Can Large Language Models Transform Computational Social Science?

By Caleb Ziems, William B. Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, Diyi Yang. International Conference on Computational Logic, 2023

564 citations recorded by the source corpus.

Large language models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify and explain social phenomena like persuasiveness and political ideology, then LLMs could augment the computational social science (CSS) pipeline in important ways. This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 25 representative English CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers’ gold references. We conclude that the performance of today’s LLMs can augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the underlying attributes of a text). In summary, LLMs are posed to meaningfully participate in social science analysis in partnership with humans.

Subfields: Artificial Intelligence, General Social Sciences

View Paper: https://www.semanticscholar.org/paper/93752cae0d4ecd2d09d6660feb3c1860af973f18

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ji-rong Wen
2. Wayne Xin Zhao
3. Zhewei Wei
4. Yankai Lin
5. Zhiyuan Liu
6. Maosong Sun
7. Yupeng Hou
8. Xiangnan He

In General Social Sciences

1. Ji-rong Wen
2. Yankai Lin
3. Wayne Xin Zhao

Citations in the corpus

• A survey on large language model based autonomous agents (2023, methodology): “[86] Jurisprudence ChatLaw [25], Blind Judgement [61] Research Assistant Ziems et al [187], Bail et al.”
• Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, unknown): “This is supported by numerous studies, such as [86], [87], which incorporate human alignment evaluation to provide a more in-depth analysis of their methods’ performance.”
• Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024, unknown): “Recent studies such as [Chen et al. , 2023b; Kaiya et al. , 2023; Li et al. , 2023a; Li et al. , 2023f; Ziems et al. , 2023] highlight the evolving complexity in multi-agent systems, LLM impacts on social networks, and their integration into social science research.”