Mitigating harm in language models with conditional-likelihood filtration

By Helen Ngo, Cooper D. Raterink, J. Ara'ujo, Ivan Zhang, Carol Chen, Adrien Morisot, Nick Frosst. 2021

51 citations recorded by the source corpus.

Language models trained on large-scale unfiltered datasets curated from the open web acquire systemic biases, prejudices, and harmful views from their training data. We present a methodology for programmatically identifying and removing harmful text from web-scale datasets. A pretrained language model is used to calculate the log-likelihood of researcher-written trigger phrases conditioned on a specific document, which is used to identify and filter documents from the dataset. We demonstrate that models trained on this filtered dataset exhibit lower propensity to generate harmful text, with a marginal decrease in performance on standard language modeling benchmarks compared to unfiltered baselines. We provide a partial explanation for this performance gap by surfacing examples of hate speech and other undesirable content from standard language modeling benchmarks. Finally, we discuss the generalization of this method and how trigger phrases which reflect specific values can be used by researchers to build language models which are more closely aligned with their values.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Training language models to follow instructions with human feedback (2022, background): “There are many ways to mitigate these harms, including by fine-tuning on a small, value-targeted dataset (Solaiman and Dennison, 2021), filtering the pretraining dataset (Ngo et al., 2021), or human-in-the-loop data collection (Dinan et al., 2019; Xu et al., 2020).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “, 2021); Removing toxic data using classifiers (Ngo et al., 2021) Privacy Filtering private/duplicate data (Henderson et al.”
A General Language Assistant as a Laboratory for Alignment (2021, methodology): “On the other end of the spectrum, a rather different technique is to ﬁlter pretraining data, as discussed in [NRA + 21].”
Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “, 2021), supervised fine-tuning on curated data (Solaiman & Dennison, 2021; Ngo et al., 2021; Welbl et al., 2021; Chung et al., 2022) or selected samples from the model (Zelikman et al.”
Low-Resource Languages Jailbreak GPT-4 (2023, background): “As large language models (LLMs) are deployed in user-facing applications like chatbots and writing tools, safety-training and red-teaming are crucial to avoid AI safety failures [27, 32, 36, 48].”
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, background): “Another proposal is to filter hazardous information from the pretraining data (Ngo et al., 2021), but adversaries may reintroduce this information through finetuning (Zhan et al., 2023; Qi et al., 2023; Pelrine et al., 2023).”
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2022, background): “Several approaches have been proposed to address these issues, such as ﬁltering unsafe text from training data (Xu et al., 2021b; Ngo et al., 2021), using specialized decoding procedures for safer generation (Liu et al., 2021), and controlling language generation (Keskar et al., 2019; Dathathri et…”