Multi-step Jailbreaking Privacy Attacks on ChatGPT

By Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Yangqiu Song. Conference on Empirical Methods in Natural Language Processing, 2023

495 citations recorded by the source corpus.

With the rapid progress of large language models (LLMs), many downstream NLP tasks can be well solved given appropriate prompts. Though model developers and researchers work hard on dialog safety to avoid generating harmful content from LLMs, it is still challenging to steer AI-generated content (AIGC) for the human good. As powerful LLMs are devouring existing text data from various domains (e.g., GPT-3 is trained on 45TB texts), it is natural to doubt whether the private information is included in the training data and what privacy threats can these LLMs and their downstream applications bring. In this paper, we study the privacy threats from OpenAI's ChatGPT and the New Bing enhanced by ChatGPT and show that application-integrated LLMs may cause new privacy threats. To this end, we conduct extensive experiments to support our claims and discuss LLMs' privacy implications.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/025ca4c125d6ecabc816a56f160e5c992abc76d9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

Large Language Models Cannot Self-Correct Reasoning Yet (2023, background): “…content have drawn significant attention from the community (Bang et al., 2023; Alkaissi & McFarlane, 2023; Zheng et al., 2023; Shi et al., 2023; Carlini et al., 2021; Huang et al., 2022; Shao et al., 2023; Li et al., 2023; Wei et al., 2023; Zhou et al., 2023b; Zou et al., 2023, inter alia ).”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “In this section, we examine the range of existing defense methods against various attacks and vulnerabilities associated with LLMs 1 .”
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (2023, background): “Attacks on open-source models Various methods have been proposed to expose the vulnerabilities of open-source LLMs. Wei et al. (2023) focus on categorizing modifications to prompts that can subvert safety tuning.”
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, background): “…undesired behavior (Gehman et al., 2020; Bommasani et al., 2021; walkerspider, 2022; Perez et al., 2022; Perez & Ribeiro, 2022; Kang et al., 2023; Li et al., 2023; Ganguli et al., 2022; Schulhoff et al., 2023; OpenAI, 2023b; Jones et al., 2023; Zou et al., 2023; Huang et al., 2024; Zeng et al.,…”
Large Language Model Alignment: A Survey (2023, background): “Prompt Attacks involve designing or searching for prompts that lead LMs to output information from the training data, including private details (Carlini et al., 2021; Lehman et al., 2021; Li et al., 2023a; Deng et al., 2023).”
Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study (2023, background): “For example, [7] introduces a multi-step jailbreaking attack against C HAT - GPT to steal private personal information, which cause severe privacy concerns.”
Jailbroken: How Does LLM Safety Training Fail? (2023, methodology): “5 via a computer security lens, and Li et al. [33], who focus on personally identifiable information (PII) extraction rather than general harm.”
"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models (2023, unknown): “Jailbreak Prompts on LLMs. Jailbreak prompts have garnered increasing attention in the academic research community recently [26, 42, 46, 86, 95].”
AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (2023, background): “Works concurrently also investigate the potential of generating jailbreak prompts from LLMs (Deng et al., 2023), jailbreak by hand-crafted multi-steps prompts (Li et al., 2023), and effectiveness of token-level jailbreaks in black-box scenarios (Lapid et al., 2023).”
Defending ChatGPT against jailbreak attack via self-reminders (2023, unknown): “44, our study assesses the efficacy of the self-reminder for an email address recovery attack against ChatGPT with and without self-reminders on the sampled 100 frequent and 100 infrequent emails from the Enron Email Dataset 45 .”
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023, background): “Most existing research on jailbreak attacks predominantly relies on manually crafting prompts [23, 31, 36, 49, 63, 64].”
StruQ: Defending Against Prompt Injection with Structured Queries (2024, unknown): “Beyond prompt injection and jail-breaking, researchers have studied other attacks on LLMs, including data extraction [52, 53, 54, 55, 56] and task-specific attacks to decrease the LLM’s performance [13, 57, 14].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “However, a similar attack is successful for instruction tuning with very few examples [Wan et al., 2023, Xu et al., 2023a], and poisoning web-scale datasets is possible under realistic assumptions [Carlini et al., 2023a].”
Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models (2025, unknown): “Here’s”’ (Mozes et al., 2023) or “Do anything now” (Mozes et al., 2023)) and implicit templates (e.g., role-playing (Bhardwaj & Poria, 2023; Shah et al., 2023), storytelling (Li et al., 2023b) and virtual scenarios (Li et al., 2023a; Kang et al., 2024; Singh et al., 2023; Du et al., 2023)).”