Fine-Tuning a Transformer-Based Language Model to Avoid Generating Non-Normative Text

By Xiangyu Peng, Siyan Li, Spencer J. Frazier, Mark O. Riedl. 2020

8 citations recorded by the source corpus.

Large-scale, transformer-based language models such as GPT-2 are pretrained on diverse corpora scraped from the internet. Consequently, they are prone to generating content that one might find inappropriate or non-normative (i.e. in violation of social norms). In this paper, we describe a technique for fine-tuning GPT-2 such that the amount of non-normative content generated is significantly reduced. A model capable of classifying normative behavior is used to produce an additional reward signal; a policy gradient reinforcement learning technique uses that reward to fine-tune the language model weights. Using this fine-tuning technique, with 24,000 sentences from a science fiction plot summary dataset, halves the percentage of generated text containing non-normative behavior from 35.1% to 15.7%.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…al. (2019); Webster et al. (2019); Ethayarajh (2020); Gaut et al. (2020); Gencoglu (2020); Hovy et al. (2020); Huang et al. (2020); Kim et al. (2020); Peng et al. (2020); Ravfogel et al. (2020); Rios (2020); Sap et al. (2020); Saunders and Byrne (2020); Sheng et al. (2020); Sweeney and Najafian…”
Large pre-trained language models contain human-like biases of what is right and wrong to do (2021, background): “hate speech is non-normative (but not all non-normative descriptions are toxic) [29], we show that the identified direction can help attenuating or even preventing the toxic degeneration in LMs.”