A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models

By Junjie Ye, Xuanting Chen, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou and 4 more. arXiv.org, 2023

490 citations recorded by the source corpus.

GPT series models, such as GPT-3, CodeX, InstructGPT, ChatGPT, and so on, have gained considerable attention due to their exceptional natural language processing capabilities. However, despite the abundance of research on the difference in capabilities between GPT series models and fine-tuned models, there has been limited attention given to the evolution of GPT series models' capabilities over time. To conduct a comprehensive analysis of the capabilities of GPT series models, we select six representative models, comprising two GPT-3 series models (i.e., davinci and text-davinci-001) and four GPT-3.5 series models (i.e., code-davinci-002, text-davinci-002, text-davinci-003, and gpt-3.5-turbo). We evaluate their performance on nine natural language understanding (NLU) tasks using 21 datasets. In particular, we compare the performance and robustness of different models for each task under zero-shot and few-shot scenarios. Our extensive experiments reveal that the overall ability of GPT series models on NLU tasks does not increase gradually as the models evolve, especially with the introduction of the RLHF training strategy. While this strategy enhances the models' ability to generate human-like responses, it also compromises their ability to solve some tasks. Furthermore, our findings indicate that there is still room for improvement in areas such as model robustness.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/98b40eb3ce79c24d9726556947e2e2094737fe46

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Sentiment Analysis in the Era of Large Language Models: A Reality Check (2023, background): “They can directly perform tasks in zero-shot or few-shot in-context learning manner and achieve strong performance without the need for any in-domain supervised training (Bang et al., 2023; Ye et al., 2023; Zhong et al., 2023; Yang et al., 2023).”
VeriGen: A Large Language Model for Verilog Code Generation (2023, unknown): “5-turbo) [31], GPT4 [17], and Google’s PALM2 [6].”
GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (2023, methodology): “It is taught using a corpus of 500 billion words called ‘‘Common Crawl’’ that was gathered from a sizable content archive and the internet [26].”
A Survey of Large Language Models (2023, unknown): “ses [47, 349], when LLMs are trained to exhibit some specific ability that allows them to excel in some areas, they might struggle in others.”