Fine-tuning language models to find agreement among humans with diverse preferences

By Michiel A. Bakker, Martin Chadwick, Hannah Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew Botvinick and 1 more. Neural Information Processing Systems, 2022

336 citations recorded by the source corpus.

Recent work in large language modeling (LLMs) has used fine-tuning to align outputs with the preferences of a prototypical user. This work assumes that human preferences are static and homogeneous across individuals, so that aligning to a a single"generic"user will confer more general alignment. Here, we embrace the heterogeneity of human preferences to consider a different challenge: how might a machine help people with diverse views find agreement? We fine-tune a 70 billion parameter LLM to generate statements that maximize the expected approval for a group of people with potentially diverse opinions. Human participants provide written opinions on thousands of questions touching on moral and political issues (e.g.,"should we raise taxes on the rich?"), and rate the LLM's generated candidate consensus statements for agreement and quality. A reward model is then trained to predict individual preferences, enabling it to quantify and rank consensus statements in terms of their appeal to the overall group, defined according to different aggregation (social welfare) functions. The model produces consensus statements that are preferred by human users over those from prompted LLMs (>70%) and significantly outperforms a tight fine-tuned baseline that lacks the final ranking step. Further, our best model's consensus statements are preferred over the best human-generated opinions (>65%). We find that when we silently constructed consensus statements from only a subset of group members, those who were excluded were more likely to dissent, revealing the sensitivity of the consensus to individual contributions. These results highlight the potential to use LLMs to help groups of humans align their values with one another.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/de1c7ae2818aa26fc86a0ea8ed70014cffc8b20a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Diffusion Model Alignment Using Direct Preference Optimization (2023, background): “Alternative approaches sample base model answers and select based on predicted rewards [4, 6, 16] to use for supervised training [3, 18, 53].”
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “Best-of-n (or re-ranking) [62, 5, 21, 8] 3: The ranking of methods trained and evaluated in AlpacaFarm matches that of meth-ods trained and evaluated in the human-based pipeline.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “Some works have begun to assess whether LLMs can be used to facilitate agreement between different humans [Bakker et al., 2022] and to codify the broad-ranging principles under which deployment of AI systems for public good can be assessed [Floridi and Cowls, 2022, Sartori and Theodorou, 2022].”
Towards Understanding Sycophancy in Language Models (2023, neutral): “Human evaluators are imperfect (Saunders et al., 2022; Gudibande et al., 2023), make mistakes e.g., due to limited time (Chmielewski & Kucker, 2020) or cognitive biases (Pandey et al., 2022), and sometimes have diverse, contradictory preferences (Bakker et al., 2022).”
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies (2022, background): “Other forms of alignment, such as Bakker et al. (2022)’s recent work on generating opinions with high consensus across heterogeneous and opposing groups, may be beneficial for creating LMs that retain realistic and demographically nuanced forms of human bias.”