Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

By Irene Solaiman, Christy Dennison. 2021

274 citations recorded by the source corpus.

Language models can generate harmful and biased outputs and exhibit undesirable behavior according to a given cultural context. We propose a Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets, an iterative process to significantly change model behavior by crafting and fine-tuning on a dataset that reflects a predetermined set of target values. We evaluate our process using three metrics: quantitative metrics with human evaluations that score output adherence to a target value, toxicity scoring on outputs; and qualitative metrics analyzing the most common word associated with a given social category. Through each iteration, we add additional training dataset examples based on observed shortcomings from evaluations. PALMS performs significantly better on all metrics compared to baseline and control models for a broad range of GPT-3 language model sizes without compromising capability integrity. We find that the effectiveness of PALMS increases with model size. We show that significantly adjusting language model behavior is feasible with a small, hand-curated dataset.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Training language models to follow instructions with human feedback (2022, background): “There are many ways to mitigate these harms, including by fine-tuning on a small, value-targeted dataset (Solaiman and Dennison, 2021), filtering the pretraining dataset (Ngo et al., 2021), or human-in-the-loop data collection (Dinan et al., 2019; Xu et al., 2020).”
Ethical and social risks of harm from Language Models (2021, methodology): “…2021; Bommasani et al., 2021; Dinan et al., 2021; Kenton et al., 2021), and papers identifying ways to mitigate potential harms (Chen et al., 2021a; Solaiman and Dennison, 2021; Welbl et al., 2021).2 For this report, we seek to build on this prior work by proposing an initial taxonomy of risks…”
TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “Models could be fine-tuned on a set of examples chosen to demonstrate truthfulness (Solaiman and Dennison, 2021) or fine-tuned by reinforcement learning from human feedback (Stiennon et al.”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “For models of O(100B) parameters, recent work has finetuned task-specific models for program synthesis (Austin et al., 2021; Chen et al., 2021), summarization (Wu et al., 2021), as well as improved bias and fairness behavior (Solaiman & Dennison, 2021).”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “A non-exhaustive categorical framework and human reviewed evaluations [228] can capture some aspects of culture.”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “Our results also support the conclusions of [55], who found that fine-tuning GPT-3 [12] on value-targeted datasets produced by prompting GPT-3 with moral positions reduced toxicity and improved human evaluation scores.”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, methodology): “To develop the list of topic tags, we referred to the taxonomies of potential harms of language models in [48, 57], industry content moderation guidelines, and a manual review of the top 100 most harmful conversations in our dataset.”
A General Language Assistant as a Laboratory for Alignment (2021, neutral): “Our use of a small amount of high-quality data for alignment is most similar to [SD21].”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “More recently, Solaimon & Dennison (2021) ﬁnd that iteratively ﬁne-tuning a model with small-scale Values-Targeted Datasets reduces the toxicity of GPT-3 (Brown et al., 2020).”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “An alternative consideration is the addition of diverse values-targeted data (Solaiman & Dennison, 2021) during pre-training in order to instill beneficial biases in a model.”
Red Teaming Language Models with Language Models (2022, background): “…we can address the specific biases by training the model to unlearn them, e.g., by finetuning on examples that align better with human values (Solaiman and Dennison, 2021) or reducing the likelihood of the current, offensive replies with unlikelihood training (Welleck et al., 2020; Li et…”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “…et al., 2021), using scorers or safety filters while decoding (Roller et al., 2021; Xu et al., 2021), supervised fine-tuning on curated data (Solaiman & Dennison, 2021; Ngo et al., 2021; Welbl et al., 2021; Chung et al., 2022) or selected samples from the model (Zelikman et al., 2022;…”