Is ChatGPT a General-Purpose Natural Language Processing Task Solver?

By Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, Diyi Yang. Conference on Empirical Methods in Natural Language Processing, 2023

872 citations recorded by the source corpus.

Spurred by advancements in scale, large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot -- i.e., without adaptation on downstream data. Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community due to the fact that it can generate high-quality responses to human input and self-correct previous mistakes based on subsequent conversations. However, it is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot. In this work, we empirically analyze the zero-shot learning ability of ChatGPT by evaluating it on 20 popular NLP datasets covering 7 representative task categories. With extensive empirical studies, we demonstrate both the effectiveness and limitations of the current version of ChatGPT. We find that ChatGPT performs well on many tasks favoring reasoning capabilities (e.g., arithmetic reasoning) while it still faces challenges when solving specific tasks such as sequence tagging. We additionally provide in-depth analysis through qualitative case studies.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/873a581320d928249609d3c07229d5af182a379c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Can Large Language Models Transform Computational Social Science? (2023, methodology): “, 2017) series, and ChatGPT (Qin et al., 2023; Gilardi et al., 2023) which is the conversation-based LLM trained through RLHF (Christiano et al.”
A Categorical Archive of ChatGPT Failures (2023, neutral): “However, for future investigations, it may be useful to focus on more specific categories of problems, such as sentiment analysis, named entity recognition, translation [27, 28], summarization [66], and language ambiguity [47], in order to gain a more detailed understanding of ChatGPT’s shortcomings”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “Different from some previous work that exploits two-stage prompting to adopt a zero-shot chain of thoughts (Kojima et al., 2022; Qin et al., 2023), we directly utilize single-stage prompting that only adds the task description D into each input X to simulate the common approach of general users for…”
ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification (2023, background): “Secondly, like some of previous studies (Qin et al., 2023; Hendy et al., 2023), we plan to extend the comparison to other GPT models, such as the GPT3.”
A Survey of Large Language Models (2023, positive): “Empirical results [41, 46, 349, 392] have shown the superior capacities of LLMs as a general-purpose task solver.”
Large Language Models as Analogical Reasoners (2023, background): “Large language models (LLMs) demonstrate strong performance across various tasks (Brown et al., 2020; Chowdhery et al., 2022; Liang et al., 2022; Qin et al., 2023).”
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023, background): “Large language models (LLMs), such as Chat-GPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023), have demonstrated unprecedented performance across various natural language processing (NLP) tasks (Qin et al., 2023) and multi-modal tasks (Shen et al., 2023).”
Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4 (2023, methodology): “After investigating the methods of promptdesigning for reasoning tasks, we find that there are mainly three types of prompt designing for NLI tasks, specifying the labels (entailment, neutral or contradiction)(Qin et al., 2023), specifying the reasoning method (induction or deduction)(Bang et al.”
Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration (2023, background): “Although large language models (LLMs) have demonstrated impressive performance as general task-solving agents, they still encounter challenges (Qin et al., 2023; Bang et al., 2023; OpenAI, 2023b; Bubeck et al., 2023) in various knowledge-intensive and reasoning-intensive tasks due to factual…”
Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study (2023, background): “and question answering, recent research has begun exploring ChatGPT’s concordance with human capabilities, encompassing language proficiency (Qin et al., 2023), fact-checking (Kasneci et al.”
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, background): “…robust complex reasoning capabilities, which should also be facilitated by the creation of more comprehensive benchmarks for assessing these abilities, such as works by Laskar et al. (2023b); Qin et al. (2023); Davis (2023a), particularly when multiple abilities are required to complete the tasks.”
A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, methodology): “In this way, RLHF can be applied for more complex or customized tasks with a well-specified reward model.”
ChatGPT and Open-AI Models: A Preliminary Review (2023, background): “empirically analyzed ChatGPT’s zero-shot learning abilities, evaluating its performance on twenty frequently used NLP datasets covering seven primary task categories [57].”
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, methodology): “5 and bert-style models [78; 110].”
Knowledge Graph Prompting for Multi-Document Question Answering (2023, background): “Due to the emergence of large language models (LLMs), the ”pre-train, prompt, predict” paradigm has revolutionized natural language processing (NLP) in real-world applications, such as open-domain question answering (O-QA), fact-checking (FC), and arithmetic reasoning (AR) (Chen et al. 2017; Asai et”
Is GPT-3 a Good Data Annotator? (2022, background): “In instances where more intricate tasks are encountered, previous research endeavors (Drozdov et al., 2023; Liu et al., 2023; Qin et al., 2023b) have demonstrated the remarkable capabilities of LLMs in complex NLP tasks, encompassing reasoning, semantic parsing, and dialogues, even in situations…”