Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT

By Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, Dacheng Tao. arXiv.org, 2023

318 citations recorded by the source corpus.

Recently, ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. Several prior studies have shown that ChatGPT attains remarkable generation ability compared with existing models. However, the quantitative analysis of ChatGPT's understanding ability has been given little attention. In this report, we explore the understanding ability of ChatGPT by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models. We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achieves comparable performance compared with BERT on sentiment analysis and question-answering tasks. Additionally, by combining some advanced prompting strategies, we show that the understanding ability of ChatGPT can be further improved.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/6839816cd3ab194dfa3ffe3066cc35d099820e00

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (2023, background): “Conventional objective metrics like perplexity [42] and accuracy [43, 44, 45, 46] on downstream tasks[24] provide valuable insights, but they may not effectively guide the choice of hyperparameters to enhance LLMs [47] because evaluating LLMs requires other subjective metrics.”
Sentiment Analysis in the Era of Large Language Models: A Reality Check (2023, background): “Although there have been some initial attempts to apply LLMs to sentiment analysis (Deng et al., 2023; Zhong et al., 2023; Wang et al., 2023), these studies are often limited to some specific tasks and adopt different models, datasets, and settings in experiments.”
ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification (2023, background): “, 2022) which was shown to significantly improve ChatGPT’s performance on classification tasks (Zhong et al., 2023).”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “Along this line, several recent studies have attempted to examine the performance of ChatGPT more comprehensively on multiple datasets (Bang et al., 2023; Qin et al., 2023; Koco’n et al., 2023; Zhong et al., 2023).”
Towards Making the Most of ChatGPT for Machine Translation (2023, background): “Chat-GPT possesses diverse abilities of NLP, including question answering, dialogue generation, code de-bugging, generation evaluation, and so on (Qin et al., 2023; Zhong et al., 2023; Wang et al., 2023a; Kocmi and Federmann, 2023; Lu et al., 2023b; Wang et al., 2023b).”
Leveraging Large Language Models for Mental Health Prediction via Online Text Data (2023, background): “Some work explored the capability of LLMs for sentiment analysis and emotion reasoning [45, 65, 94].”
Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, methodology): “5 model [72], especially in the question-answering task, where its performance is comparable to bert-style models [104].”
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, methodology): “In other tasks, according to literature [78], ChatGPT performs well in natural language inference, i.e., the task of inferring sentence relationships, and its performance on this task is significantly better than all bert-style models [110].”
Is ChatGPT a Good NLG Evaluator? A Preliminary Study (2023, background): “For example, Zhong et al. (2023) show that Chat-GPT can attain the comparable understanding abil-a r X i v : 2303 .”
ChatGPT and Open-AI Models: A Preliminary Review (2023, methodology): “The GLUE benchmark tool was called on to examine ChatGPT’s responses to various NLU tasks, such as linguistic accept-Future ability, sentiment analysis, paraphrasing, textual similarity, and more.”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, methodology): “For example, ChatGPT’s performance on the GLUE STS-B dataset, which is a regression task evaluating sentence similarity, is inferior to a fine-tuned RoBERTa performance [141].”
The Price of Prompting: Profiling Energy Use in Large Language Models Inference (2024, unknown): “In the current literature, several endeavors aim to quantify and mitigate the environmental impact of AI.”