Reducing Non-Normative Text Generation from Language Models

By Xiangyu Peng, Siyan Li, Spencer J. Frazier, Mark O. Riedl. 2020

33 citations recorded by the source corpus.

Large-scale, transformer-based language models such as GPT-2 are pretrained on diverse corpora scraped from the internet. Consequently, they are prone to generating non-normative text (i.e. in violation of social norms). We introduce a technique for fine-tuning GPT-2, using a policy gradient reinforcement learning technique and a normative text classifier to produce reward and punishment values. We evaluate our technique on five data sets using automated and human participant experiments. The normative text classifier is 81-90% accurate when compared to gold-standard human judgements of normative and non-normative generated text. Our normative fine-tuning technique is able to reduce non-normative text by 27-61%, depending on the data set.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yejin Choi
2. Nanyun Peng
3. Kai-Wei Chang
4. Emily Sheng
5. Ximing Lu
6. Sean Welleck
7. Yulia Tsvetkov
8. Premkumar Natarajan

In Natural Language Processing

1. Yejin Choi
2. Nanyun Peng
3. Kai-Wei Chang
4. Emily Sheng
5. Ximing Lu
6. Premkumar Natarajan
7. Sean Welleck
8. Mark O. Riedl

In Safety, Robustness & Interpretability

1. Nanyun Peng
2. Emily Sheng
3. Kai-Wei Chang
4. Yejin Choi
5. Premkumar Natarajan
6. Patrick Schramowski
7. Ximing Lu
8. Kristian Kersting

Citations in the corpus

• Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “…2019) There are also metrics for other bias evaluation setups in continuation generation tasks involving sentiment (Shwartz et al., 2020), the ratio of gendered words (Solaiman et al., 2019; Vig et al., 2020; Dinan et al., 2020a), and other novel metrics (Peng et al., 2020; Yeo and Chen, 2020).”
• Quark: Controllable Text Generation with Reinforced Unlearning (2022, positive): “Some recent works use RL for post-hoc modification of language models, e.g., unlearning toxicity [14] or non-normative generations [55].”
• Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning (2022, methodology): “Work by (Peng et al., 2020) trained a language model to generate less non-normative language using a normative text classifier (Frazier et al., 2020) to modify the loss function.”
• Large pre-trained language models contain human-like biases of what is right and wrong to do (2021, background): “Approaches have been developed to decrease the level of bias in these models [21, 22] and to prevent the toxic degeneration in language models [23, 24, 25].”
• The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems (2022, unknown): “…(Ficler and Goldberg, 2017; Gehman et al., 2020), safety classifiers (Xu et al., 2021), controlled language generation (Keskar et al., 2019; Ziegler et al., 2019; Luketina et al., 2019), and re-inforcement learning (Peng et al., 2020; Liu et al., 2021a; Ziegler et al., 2019; Luketina et al., 2019).”
• Debiasing Pre-Trained Language Models via Efficient Fine-Tuning (2022, unknown): “Finally, Peng et al. (2020) applied normative fine-tuning on GPT-2 to reduce the frequency of non-normative output.”