ChatGPT: Jack of all trades, master of none

By Jan Koco'n, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz and 10 more. Information Fusion, 2023

765 citations recorded by the source corpus.

OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT) and revolutionized the approach in artificial intelligence to human-model interaction. Several publications on ChatGPT evaluation test its effectiveness on well-known natural language processing (NLP) tasks. However, the existing studies are mostly non-automated and tested on a very limited scale. In this work, we examined ChatGPT's capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans, such as sentiment analysis, emotion recognition, offensiveness, and stance detection. In contrast, the other tasks require more objective reasoning like word sense disambiguation, linguistic acceptability, and question answering. We also evaluated GPT-4 model on five selected subsets of NLP tasks. We automated ChatGPT and GPT-4 prompting process and analyzed more than 49k responses. Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation. For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT. We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss. It especially refers to pragmatic NLP problems like emotion recognition. We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions. Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI. Our results provide the basis for a fundamental discussion of whether the high quality of recent predictive NLP models can indicate a tool's usefulness to society and how the learning and validation procedures for such systems should be established.

Subfields: Artificial Intelligence, Health Informatics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/5848737f78397f72ceae2ba6f3419a6a8502b8ba

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Diyi Yang
2. Jiaao Chen
3. Jie Zhou
4. Zhixu Li
5. Omar Shaikh
6. Caleb Ziems
7. Siyuan Ma
8. Anind K. Dey

In Health Informatics

1. Siyuan Ma
2. Yiheng Liu
3. Antong Li
4. Zheng Liu
5. Haoyang He
6. Mengshen He
7. Dajiang Zhu
8. Tianle Han

Citations in the corpus

• A Survey of Large Language Models (2023, neutral): “Furthermore, several studies have also shown that LLMs may misunderstand unfamiliar concepts [392, 393] on information extraction tasks from specific domains, and face challenges in solving pragmatic emotionrelated tasks [391] (e.”
• Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, neutral): “When the model satisfies safety constraints, Safe RLHF employs a smaller λ to preserve harmlessness, thereby avoiding over-optimization, as illustrated in Figure 6c.”
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study (2023, background): “5, ChatGPT and GPT-4), which are performed through chatting (or asking) with it, have obtained promising results on various natural language understanding (NLU) and natural language generation (NLG) downstream tasks (Ouyang et al., 2022; Koco ´ n et al., 2023; Qin et al., 2023; Huang et al., 2023; Y”
• On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, neutral): “Selective annotation represents an optimal tradeoff between expensive and precise human annotation and economic but relatively rough LLMs-based annotation(Wang et al., 2021; Kocon et al., 2023).”
• A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models (2023, neutral): “For example, Koco’n et al. (2023) compared the performance of ChatGPT with state-of-the-art models on 25 different NLP tasks, revealing certain biases and shortcomings of ChatGPT.”
• Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, background): “However, according to literature [40], ChatGPT has losses not exceeding 25% on most tasks, except for three relatively subjective emotion perception tasks where it performs poorly.”
• Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, background): “In terms of stability, reference [43] concludes that ChatGPT’s performance is always lower than SOTA, the current state-of-the-art model, in almost all tasks.”
• Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs (2024, background): “…models being consistently better than open-source ones, we note that (1) this is often driven by hype, while there is evidence of the opposite (Koco´n et al., 2023), (2) research done solely on closed LLMs limits scientific progress, bringing benefits mainly to the LLM vendors and (3) LLM…”
• Can Large Language Models Transform Computational Social Science? (2023, methodology): “While the use of a single set of instructions is common in recent broad LLM benchmarks (Liang et al., 2022; Kocoń et al., 2023; Qin et al., 2023), it does not capture instruction-based variance as we discussed further in §7.7 (Zhao et al., 2021).”
• Leveraging Large Language Models for Mental Health Prediction via Online Text Data (2023, background): “Some work explored the capability of LLMs for sentiment analysis and emotion reasoning [45, 65, 94].”