Text Detoxification using Large Pre-trained Neural Models

By David Dale, Anton Voronov, Daryna Dementieva, Varvara Logacheva, Olga Kozlova, Nikita Semenov, Alexander Panchenko. 2021

100 citations recorded by the source corpus.

We present two novel unsupervised methods for eliminating toxicity in text. Our first method combines two recent ideas: (1) guidance of the generation process with small style-conditional language models and (2) use of paraphrasing models to perform style transfer. We use a well-performing paraphraser guided by style-trained language models to keep the text content and remove toxicity. Our second method uses BERT to replace toxic words with their non-offensive synonyms. We make the method more flexible by enabling BERT to replace mask tokens with a variable number of words. Finally, we present the first large-scale comparative study of style transfer models on the task of toxicity removal. We compare our models with a number of methods for style transfer. The models are evaluated in a reference-free way using a combination of unsupervised style transfer metrics. Both methods we suggest yield new SOTA results.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hao Sun
2. Zhexin Zhang
3. Minlie Huang
4. Jiawen Deng
5. Jiale Cheng
6. Xiaoyan Zhu
7. Erik Cambria

In Natural Language Processing

1. Zhexin Zhang
2. Hao Sun
3. Minlie Huang
4. Jiawen Deng
5. Jiale Cheng

Citations in the corpus

• Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Similarly, text style transfer or rephrasing from toxicity to non-toxicity can also be plugged in this stage (Dale et al., 2021; Laugier et al., 2021).”
• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “The study first assessed the question-answering performance of these LLMs on Cisco certifications with varying difficulty levels, then examined their abilities in solving CTF challenges.”