A Holistic Approach to Undesired Content Detection in the Real World

By Todor Markov, S. Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, L. Weng. AAAI Conference on Artificial Intelligence, 2022

468 citations recorded by the source corpus.

We present a holistic approach to building a robust and useful natural language classification system for real-world content moderation. The success of such a system relies on a chain of carefully designed and executed steps, including the design of content taxonomies and labeling instructions, data quality control, an active learning pipeline to capture rare events, and a variety of methods to make the model robust and to avoid overfitting. Our moderation system is trained to detect a broad set of categories of undesired content, including sexual content, hateful content, violence, self-harm, and harassment. This approach generalizes to a wide range of different content taxonomies and can be used to create high-quality content classifiers that outperform off-the-shelf models.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/b69aa32ee52f5efe8e3196114581ac610da8a2b2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Multi-step Jailbreaking Privacy Attacks on ChatGPT (2023, background): “Markov et al. (2023) built a holistic system for content detection to avoid undesired content from hate speech to harmful content.”
OpenAI o1 System Card (2024, unknown): “Additionally, o1 inherits our earlier safety mitigations of training in refusal behavior for harmful requests and using moderation models for the most egregious content [14, 15].”
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations (2023, methodology): “For all experiments, we use the area under the precision-recall curve ( AUPRC ) as our evaluation metrics, following (Markov et al., 2023).”
Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “A framework has been proposed named VistaGPT to deal with the problems caused by information barriers from heterogeneity at both system and module levels in a wide range of heterogeneous vehicle automation systems [202].”
"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models (2023, unknown): “In this section, our evaluation centers on three specific external safeguards, including OpenAI moderation endpoint [48], OpenChatKit moderation model [80], and NeMo-Guardrails [57].”
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks (2023, methodology): “…(i) state-of-the-art LLMs have dramatically improved in their instruction-following ability (Ouyang et al., 2022; Wei et al., 2021; Sanh et al., 2021; Iyer et al., 2022); and (ii) Zellers et al., 2019; Markov et al., 2022; Pavlopoulos et al., 2020; Gehman et al., 2020; Guo et al., 2022).”
Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (2023, unknown): “However, text-based attacks can be easily spotted by automated keyword or perplexity-based filters, leading to security patches and, consequently, diminishing their effectiveness as a persistent threat (Greshake et al., 2023; Markov et al., 2023).”
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023, methodology): “❷ Moderation : We utilize OpenAI’s moderation API [40] to evaluate whether the response content complies with OpenAI’s usage policies.”
ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health (2023, background): “Addressing the alignment problem for LLMs is an ongoing area of research and OpenAI developed a moderation system, trained to detect a broad set of categories of undesired content, including sexual and hateful content, violence, and other controversial topics (17).”
LLM Censorship: A Machine Learning Challenge or a Computer Security Problem? (2023, methodology): “For example, Markov et al. [30] deemed impermissible strings to be those which contain content pertaining to one of multiple sensitive subjects including sexual content, violence, or self-harm.”
FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (2023, background): “Content moderation is deployed as the last ditch to prevent toxic content from reaching the end-users [30].”
LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (2023, methodology): “We utilize the OpenAI moderation API 4 (Markov et al., 2023) to tag all conversations.”
Ignore Previous Prompt: Attack Techniques For Language Models (2022, unknown): “We released the code for P ROMPT I NJECT intending to facilitate future research for the community and welcome any researcher to expand the work presented in this paper, hoping that ultimately this will lead to safer and robust use of language models in product applications.”
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, unknown): “A variety of best practices and defenses have been adopted by leading LLM developers to address malicious use, including red teaming, filters, and refusal mechanisms (Ganguli et al., 2022; Markov et al., 2023; Achiam et al., 2023; Tou-vron et al., 2023).”
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, methodology): “Effective content moderation is essential to mitigating these risks by filtering out undesired inputs [24], monitoring harmful model completions [16], and evaluating effectiveness of model safeguards by measuring refusal rates on harmful versus benign prompts [30].”
Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models (2025, unknown): “…as it on the input side; 2) another LLM to answer whether the output is harmful (Phute et al., 2023; Inan et al., 2023; Wang et al., 2023), which doubles the cost; 3) alternatively, text classifiers to do this (He et al., 2021; Kim et al., 2023; Markov et al., 2023), which is more cost-effective.”