How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection

By Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, Yupeng Wu. 2023

882 citations recorded by the source corpus.

The introduction of ChatGPT has garnered widespread attention in both academic and industrial communities. ChatGPT is able to respond effectively to a wide range of human questions, providing fluent and comprehensive answers that significantly surpass previous public chatbots in terms of security and usefulness. On one hand, people are curious about how ChatGPT is able to achieve such strength and how far it is from human experts. On the other hand, people are starting to worry about the potential negative impacts that large language models (LLMs) like ChatGPT could have on society, such as fake news, plagiarism, and social security issues. In this work, we collected tens of thousands of comparison responses from both human experts and ChatGPT, with questions ranging from open-domain, financial, medical, legal, and psychological areas. We call the collected dataset the Human ChatGPT Comparison Corpus (HC3). Based on the HC3 dataset, we study the characteristics of ChatGPT's responses, the differences and gaps from human experts, and future directions for LLMs. We conducted comprehensive human evaluations and linguistic analyses of ChatGPT-generated content compared with that of humans, where many interesting results are revealed. After that, we conduct extensive experiments on how to effectively detect whether a certain text is generated by ChatGPT or humans. We build three different detection systems, explore several key factors that influence their effectiveness, and evaluate them in different scenarios. The dataset, code, and models are all publicly available at https://github.com/Hello-SimpleAI/chatgpt-comparison-detection.

Subfields: Artificial Intelligence, Health Informatics, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Ali Borji

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

The False Promise of Imitating Proprietary LLMs (2023, methodology): “• HC3 (Guo et al., 2023): we use the ChatGPT responses from the English Human-ChatGPT Comparison Corpus.”
Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech (2023, background): “most powerful text generation model, not surprisingly, researchers and practitioners have been evaluating its capability for various tasks, including question-and-answer in financial, medical, legal, and psychological areas [10], medical report simplification [13], bug fixing in computer programs [1”
On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective (2023, background): “There are reﬂections and discussions from law [10], education [17, 22, 26, 44], human-computer interaction [45], medicine [21], and writing [5].”
Benchmarking Large Language Models in Retrieval-Augmented Generation (2023, background): “Although these models have shown remarkable general abilities (Bang et al. 2023; Guo et al. 2023), they still suffer severely from challenges including factual hallucination (Cao et al. 2020; Raunak, Menezes, and Junczys-Dowmunt 2021; Ji et al. 2023), knowledge out-dating (He, Zhang, and Roth…”
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, background): “Guo et al. (2023) conducts linguistic analyses of ChatGPT’s writing against human experts and found that ChatGPT responses are strictly focused on the given question, more formal, objective, and less emotional.”
Zero-Shot Information Extraction via Chatting with ChatGPT (2023, methodology): “For example, (Zhang et al., 2022) use ChatGPT achieved state-of-the-art performance on Stance Detection, (Guo et al., 2023) evaluated its helpfulness on question answering, (Jiao et al., 2023) state that it is a good translator for spoken language.”
OLMo: Accelerating the Science of Language Models (2024, methodology): “The evaluation suite (Wang et al., 2023; Ivison et al., 2023) for adapted models.”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “…many traditional benchmarks (Sun et al., 2023; Sobania et al., 2023; Zhang et al., 2023a; Dhingra et al., 2023; i.a. ), and surpass human performance on tasks that require seemingly nontrivial reasoning (Chowdhery et al., 2022; Hoffmann et al., 2022; Malinka et al., 2023; Guo et al., 2023; i.a. ).”
ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification (2023, methodology): “Secondly, like some of previous studies (Qin et al., 2023; Hendy et al., 2023), we plan to extend the comparison to other GPT models, such as the GPT3.”
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, neutral): “The game-changing emergence of Large Language Models (LLMs) instigated a significant paradigm shift in the field of deep learning (Zhang et al., 2023a; Guo et al., 2023; Bang et al., 2023).”
Uncovering ChatGPT’s Capabilities in Recommender Systems (2023, background): “These LLMs possess a versatile nature and extensive world knowledge, allowing them to be applied not only in various NLP tasks [3, 5, 16], but also in domains such as education [11, 19, 22], medicine [1, 2, 29], search [20, 26, 37] and law [4, 8, 38].”
A Categorical Archive of ChatGPT Failures (2023, background): “ChatGPT lacks human-like divergences and tends to be overly literal, leading to misses in some cases [15].”
Unveiling Security, Privacy, and Ethical Concerns of ChatGPT (2023, background): “[33], analyzed linguistic features in both English and Chinese texts.”
A large-scale comparison of human-written versus ChatGPT-generated essays (2023, neutral): “Guo et al. 14 create a set of free-text question answering tasks based on data they collected from the internet, e.g. question answering from Reddit.”
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text (2024, neutral): “Our coverage of the landscape is non-exhaustive, and we refer to recent surveys Ghosal et al. (2023); Tang et al. (2023); Dhaini et al. (2023); Guo et al. (2023) for additional details.”
DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text (2023, background): “, 2019) on the output of GPT2, while the ChatGPT Detector (Guo et al., 2023) fine-tunes RoBERTa on the HC3 (Guo et al.”
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023, methodology): “Through RLHF training, ChatGPT has been observed to have impressive capabilities in various aspects, including generating high-quality responses to human input, rejecting inappropriate questions, and self-correcting previous errors based on subsequent conversations [Guo et al., 2023].”