Model Alignment as Prospect Theoretic Optimization

By Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, Douwe Kiela. International Conference on Machine Learning, 2024

1,067 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/942bb109cfcaef46e2d2e5573bed0db764be657b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, methodology): “Preference tuning further improves responses of LLMs using reinforcement learning human feedback (RLHF) [5] or preference optimization [2, 19, 23, 44] based on a preference dataset.”
Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, unknown): “A large body of work has examined improved methods for alignment (Rafailov et al., 2023; Ethayarajh et al., 2024; Zou et al., 2023a; Bai et al., 2022b; Ouyang et al., 2022; Touvron et al., 2023b; Team et al., 2024).”
OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework (2024, methodology): “Furthermore, OpenRLHF implements multiple alignment algorithms, including Direct Preference Optimization (DPO) [20], Kahneman-Tversky optimization (KTO) [10], conditional SFT [12], and rejection sampling [28], providing an accessible full-scale RLHF training framework.”
Process Reinforcement through Implicit Rewards (2025, unknown): “Implicit rewards are broadly adopted in LLM alignment (Rafailov et al., 2023; Chen et al., 2024b; Azar et al., 2024; Ethayarajh et al., 2024; Rosset et al., 2024; Chen et al., 2024a).”
Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning (2024, methodology): “Following the proof of t = 1 case and using the monotonicity property of { ∆ NPO ,i } Kt =0 , we have ) by the definition of pred i and the monotonicity of { b ( t )NPO ,i } Kt =0 , it follows from Claim (12) that Putting the last two displays together and following the same argument as the t = 1 ca”
SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, unknown): “KTO [29] learns from non-paired preference data.”
ORPO: Monolithic Preference Optimization without Reference Model (2024, methodology): “Ethayarajh et al. (2024) and Cai et al. (2023) proposed Kahneman-Tversky Optimisation (KTO) and Unified Language Model Alignment (ULMA) that does not require the pair-wise preference dataset, unlike RLHF and DPO.”
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, unknown): “We also include trainers of the advanced preference optimization methods such as KTO (Ethayarajh et al., 2024) and ORPO (Hong et al., 2024)”
Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (2025, unknown): “Offline meth-ods like DPO (Rafailov et al., 2023), SimPO (Meng et al., 2024), and KTO (Ethayarajh et al., 2024) optimize policies using pre-collected preferences, typically from human annotators or LLMs.”
S-SPPO: Semantic-Calibrated Self-Play Preference Optimization (2026, unknown): “Most prior DPO variants operate in an offline setting (Rafailov et al., 2023; Ethayarajh et al., 2024; Hong et al., 2024; Meng et al., 2024), optimizing solely on a fixed and pre-collected preference dataset.”
RLHF Workflow: From Reward Modeling to Online RLHF (2024, background): “This choice is similar to Hoang Tran (2024); Pace et al. (2024); Yuan et al. (2024b); Xu et al. (2024).”
Binary Classifier Optimization for Large Language Model Alignment (2024, unknown): “Counter to this trend, a recent work called Kahneman-Tversky Optimization (KTO) has been proposed (Ethayarajh et al., 2024).”
Evaluating Alignment Techniques for Enhancing LLM Performance in a Closed-Domain Application: A RAG Bench-Marking Study (2024, unknown): “Kahneman-Tversky Optimization (KTO) [6] leverages principles from prospect theory to align LLMs with human preferences more effectively than traditional supervised fine-tuning or preference-based reinforcement learning.”