On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective

By Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Weirong Ye, Xiubo Geng, Binxing Jiao and 2 more. IEEE Data Engineering Bulletin, 2023

310 citations recorded by the source corpus.

ChatGPT is a recent chatbot service released by OpenAI and is receiving increasing attention over the past few months. While evaluations of various aspects of ChatGPT have been done, its robustness, i.e., the performance to unexpected inputs, is still unclear to the public. Robustness is of particular concern in responsible AI, especially for safety-critical applications. In this paper, we conduct a thorough evaluation of the robustness of ChatGPT from the adversarial and out-of-distribution (OOD) perspective. To do so, we employ the AdvGLUE and ANLI benchmarks to assess adversarial robustness and the Flipkart review and DDXPlus medical diagnosis datasets for OOD evaluation. We select several popular foundation models as baselines. Results show that ChatGPT shows consistent advantages on most adversarial and OOD classification and translation tasks. However, the absolute performance is far from perfection, which suggests that adversarial and OOD robustness remains a significant threat to foundation models. Moreover, ChatGPT shows astounding performance in understanding dialogue-related texts and we find that it tends to provide informal suggestions for medical tasks instead of definitive answers. Finally, we present in-depth discussions of possible research directions.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/5c7353fac22a8fdc43fc2f5c006b5d6902c47e75

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiliang Tang
2. Xia Hu
3. Dawei Yin
4. Shirui Pan
5. Xiao Yang
6. Thien Huu Nguyen
7. Hui Liu
8. Chao Du

Citations in the corpus

• Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, unknown): “The robustness problems of LLM agent simulation can be classi fi ed into two main scenarios, adversarial attack and out-of-distribution generalization, which fundamentally stem from the robustness issues of the underlying language models (Wang et al., 2023a).”
• StruQ: Defending Against Prompt Injection with Structured Queries (2024, unknown): “Beyond prompt injection and jail-breaking, researchers have studied other attacks on LLMs, including data extraction [52, 53, 54, 55, 56] and task-specific attacks to decrease the LLM’s performance [13, 57, 14].”
• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, methodology): “Similarly, ChatGPT exhibits consistent advantages on most adversarial and OOD classification and translation tasks [117].”
• Tool Learning with Foundation Models (2023, unknown): “…reasoning additionally considers feedback from the user and environment (Figure 5), and is thus better suited to complex tasks (Madaan et al., 2023; Wang et al., 2023b), such as multi-step QA and embodied learning, where decision-making at each step is dependent on the preceding context.”
• Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “…baseline defense techniques, such as training sample filtering, reorganization, and rephrasing, have been proposed in [100]), handling adversarial inputs remains challenging for LLMs. Wang et al. conduct a thorough evaluation of the efficacy of ChatGPT from the adversarial and OOD aspects [101].”
• On Evaluating Adversarial Robustness of Large Vision-Language Models (2023, background): “[88] Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, et al.”
• Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “This raises another issue that LLMs trained on general corpus might not be able to generalize well to specific domains or new knowledge due to the lack of domain-specific knowledge or new training data [18].”
• Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs (2023, background): “Wrong predictions made by LLMs are sometimes also reasonable.”
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “For example, (Wang et al., 2023b) explores the robustness of ChatGPT under the adversarial and out-of-domain learning settings while (Jang and Lukasiewicz, 2023) examines the logical prediction consistency of ChatGPT for inputs with semantic equivalence, logical negation, or symmetricity.”