Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models

By Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, M. Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, Bryan Catanzaro. 2022

90 citations recorded by the source corpus.

Pre-trained language models (LMs) are shown to easily generate toxic language. In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models. We conduct this study on three dimensions: training corpus, model size, and parameter efficiency. For the training corpus, we propose to leverage the generative power of LMs and generate nontoxic datasets for domain-adaptive training, which mitigates the exposure bias and is shown to be more data-efficient than using a curated pre-training corpus. We demonstrate that the self-generation method consistently outperforms the existing baselines across various model sizes on both automatic and human evaluations, even when it uses a 1/3 smaller training corpus. We then comprehensively study detoxifying LMs with parameter sizes ranging from 126M up to 530B (3x larger than GPT-3), a scale that has never been studied before. We find that i) large LMs have similar toxicity levels as smaller ones given the same pre-training corpus, and ii) large LMs require more endeavor to detoxify. We also explore parameter-efficient training methods for detoxification. We demonstrate that adding and training adapter-only layers in LMs not only saves a lot of parameters but also achieves a better trade-off between toxicity and perplexity than whole model adaptation for the large-scale models.

Subfields: Large Language Models, Safety, Robustness & Interpretability, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Safety, Robustness & Interpretability

In AI Ethics, Fairness & Society

Citations in the corpus

Jailbroken: How Does LLM Safety Training Fail? (2023, methodology): “These methods can be used alongside filtering [52, 50, 38] and scrubbing the training data [40, 34].”
A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily (2023, methodology): “The data level includes pre-training data filtering (Xu et al., 2020; Welbl et al., 2021; Wang et al., 2022a), which filters out potential unsafe texts in the data through heuristics or classifiers.”
Factuality Enhanced Language Models for Open-Ended Text Generation (2022, background): “…3, generation factuality does improve with the scaling of model size, e.g., NE E R drops from 63.99% to 33.30% when parameter size scales up from 126M to 530B. Prompt Type Prompts provided to the LM are known to signiﬁcantly affect the quality and characteristics of LM continuations [61, 74, 75].”
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, background): “Inspired by the recent success of chain-of-thought that uses a simple prompt such as “let’s think step by step" (Wei et al., 2022; Kojima et al., 2022), we hypothesize that the prompt “You are an expert on Cipher Code.” in SelfCipher plays a similar role.”