How is ChatGPT's behavior changing over time?

By Lingjiao Chen, Matei A. Zaharia, James Y. Zou. Special Issue 5: Grappling With the Generative AI Revolution, 2023

676 citations recorded by the source corpus.

GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services. However, when and how these models are updated over time is opaque. Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3) opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating code, 6) US Medical License tests, and 7) visual reasoning. We find that the performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time. For example, GPT-4 (March 2023) was reasonable at identifying prime vs. composite numbers (84% accuracy) but GPT-4 (June 2023) was poor on these same questions (51% accuracy). This is partly explained by a drop in GPT-4's amenity to follow chain-of-thought prompting. Interestingly, GPT-3.5 was much better in June than in March in this task. GPT-4 became less willing to answer sensitive questions and opinion survey questions in June than in March. GPT-4 performed better at multi-hop questions in June than in March, while GPT-3.5's performance dropped on this task. Both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June than in March. We provide evidence that GPT-4's ability to follow user instructions has decreased over time, which is one common factor behind the many behavior drifts. Overall, our findings show that the behavior of the"same"LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLMs.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/e51d38bbabca719cd9c845d1e004c724e9cbdfa4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

Extracting accurate materials data from research papers with conversational language models and prompt engineering (2023, background): “Their default versions also tend to change their performance over time [38], which we overcome by using version snapshots (see Sec.”
Testing theory of mind in large language models and humans (2024, result): “This finding is consistent with longitudinal evidence that GPT models have become more reluctant to answer opinion questions over time 39 .”
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023, background): “• Adaptability: LLMs are continuously evolving, with new versions and updates being released regularly [13].”
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research (2023, positive): “Similar to the difficulties we found when using Perspective, previous work has shown the lack of reproducibility in general use text generation APIs (Ruis et al., 2022; Chen et al., 2023).”
Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024, neutral): “For example, the New York State Bar Association’s AI Task Force states that lawyers “have a duty to understand the benefits, risks and ethical implications” associated with the tools that they use (2024, 57); similarly, the State Bar of California’s Standing Committee on Professional Responsibility…”
Low-Resource Languages Jailbreak GPT-4 (2023, background): “[10] Lingjiao Chen, Matei Zaharia, and James Zou.”
Generative AI (2023, neutral): “For both open-source and closed-source models, unexpected deterioration of model performance over time highlights the need for continuous model monitoring (Chen et al. 2023).”
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023, background): “However, depending on such proprietary LMs can raise API costs and diminish reproducibility (Chen et al., 2023).”