ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification

By Taja Kuzman, Igor Mozetic, Nikola Ljubesic. 2023

110 citations recorded by the source corpus.

ChatGPT has shown strong capabilities in natural language generation tasks, which naturally leads researchers to explore where its abilities end. In this paper, we examine whether ChatGPT can be used for zero-shot text classification, more specifically, automatic genre identification. We compare ChatGPT with a multilingual XLM-RoBERTa language model that was fine-tuned on datasets, manually annotated with genres. The models are compared on test sets in two languages: English and Slovenian. Results show that ChatGPT outperforms the fine-tuned model when applied to the dataset which was not seen before by either of the models. Even when applied on Slovenian language as an under-resourced language, ChatGPT's performance is no worse than when applied to English. However, if the model is fully prompted in Slovenian, the performance drops significantly, showing the current limitations of ChatGPT usage on smaller languages. The presented results lead us to questioning whether this is the beginning of an end of laborious manual annotation campaigns even for smaller languages, such as Slovenian.

Subfields: Artificial Intelligence, Health Informatics, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Natural Language Processing

Citations in the corpus

ChatGPT outperforms crowd workers for text-annotation tasks (2023, background): “While a few studies suggested that ChatGPT might perform text annotation tasks of the kinds we have described (Kuzman et al., 2023; Huang et al., 2023), to the best of our knowledge no systematic evaluation has been conducted yet.”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “, 2023), text classification (Kuzman et al., 2023; Amin et al., 2023), grammatical error detection (Fang et al.”
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, methodology): “Kuzman et al. [46] employed ChatGPT for automated genre recognition, with the goal of simplifying the text classification task by utilizing ChatGPT’s zero-shot classification capability.”
Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, methodology): “Kuzman et al. [43] employed ChatGPT for automatic genre recognition, with the goal of simplifying the text classiﬁcation task by utilizing ChatGPT’s zero-shot classiﬁcation capability.”