Detecting Language Model Attacks with Perplexity

By Gabriel Alon, Michael Kamfonas. arXiv.org, 2023

481 citations recorded by the source corpus.

A novel hack involving Large Language Models (LLMs) has emerged, leveraging adversarial suffixes to trick models into generating perilous responses. This method has garnered considerable attention from reputable media outlets such as the New York Times and Wired, thereby influencing public perception regarding the security and safety of LLMs. In this study, we advocate the utilization of perplexity as one of the means to recognize such potential attacks. The underlying concept behind these hacks revolves around appending an unusually constructed string of text to a harmful query that would otherwise be blocked. This maneuver confuses the protective mechanisms and tricks the model into generating a forbidden response. Such scenarios could result in providing detailed instructions to a malicious user for constructing explosives or orchestrating a bank heist. Our investigation demonstrates the feasibility of employing perplexity, a prevalent natural language processing metric, to detect these adversarial tactics before generating a forbidden response. By evaluating the perplexity of queries with and without such adversarial suffixes using an open-source LLM, we discovered that nearly 90 percent were above a perplexity of 1000. This contrast underscores the efficacy of perplexity for detecting this type of exploit.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/b8b8d5655df1c6a71bbb713387863e34cc055332

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs (2024, unknown): “Specifically, we focus on mutation-based and detection-based defenses, and deliberately omit perplexity-based methods (Alon and Kamfonas, 2023; Jain et al., 2023), which depend on identifying unusually high perplexity, since our generated PAPs are human-readable with low perplexity in nature.”
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, unknown): “Definition A.1 ( k -unstable) .”
Improving Alignment and Robustness with Circuit Breakers (2024, unknown): “Inference-time defenses, such as perplexity filters [1, 26], x s ∼ D s , x r ∼ D r ▷ Sample Batch Elements 3: Loss to be Optimized 7: end for are effective only against non-adaptive attacks [36], while erase-and-check and SmoothLLM [56] incur high computational costs.”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, background): “Detection-based defenses [11,25,41,44,49] aim to detect whether the data of the target task is compromised, i.e., includes injected instruction/data or not.”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, background): “Conversely, test-time defenses like SmoothLLM (Robey et al., 2023; Ji et al., 2024) and perplexity filtering (Jain et al., 2023; Alon and Kamfonas, 2023) define wrappers around LLMs to detect potential jailbreaks.”
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (2024, unknown): “Specifically, we consider two types of defense: a) Perplexity Filter [2], which evaluates the perplexity of the input query and filters out those larger than a threshold; and b) Rephrasing Defense [15], which rephrases the original query to obtain a query that shares the same semantic meaning as…”
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, unknown): “3) Harmless: While existing defense methods, particularly ﬁlter-based [16] and detection-based [19], [38], are known to may cause unaffordable false positive cases that reject benign prompts, our proposed ICD does not have this concern.”
FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (2023, methodology): “As the prompt generated by the gradient-based approach can be easily detected by the perplexity-based filter [1], [44], we manually craft and hardcode the incitement prompt as Prompt 1, which also demonstrates the simplicity of FigStep .”
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, unknown): “PPL detection (Alon & Kamfonas, 2023; Jain et al., 2023).”
Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “Overview of LLM architecture and workflow”
Jailbreaking Black Box Large Language Models in Twenty Queries (2023, unknown): “Similar to social engineering attacks, we allow the attacker LLM A to use previous prompts and responses to iteratively refine the attack.”
Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models (2025, unknown): “On the input side, there are several guard measures: 1) Detecting suspicious patterns (Alon & Kamfonas, 2023; Jain et al., 2023), which tends to yield false positives; 2) Re-minding LLMs to align values with system prompts (Wei et al., 2023; Xie et al., 2023; Zhang et al., 2024), which can be…”