Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation

By Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, Young Jin Kim. International Conference on Machine Learning, 2024

477 citations recorded by the source corpus.

Moderate-sized large language models (LLMs) -- those with 7B or 13B parameters -- exhibit promising machine translation (MT) performance. However, even the top-performing 13B LLM-based translation models, like ALMA, does not match the performance of state-of-the-art conventional encoder-decoder translation models or larger-scale LLMs such as GPT-4. In this study, we bridge this performance gap. We first assess the shortcomings of supervised fine-tuning for LLMs in the MT task, emphasizing the quality issues present in the reference data, despite being human-generated. Then, in contrast to SFT which mimics reference translations, we introduce Contrastive Preference Optimization (CPO), a novel approach that trains models to avoid generating adequate but not perfect translations. Applying CPO to ALMA models with only 22K parallel sentences and 12M parameters yields significant improvements. The resulting model, called ALMA-R, can match or exceed the performance of the WMT competition winners and GPT-4 on WMT'21, WMT'22 and WMT'23 test datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ebd1c04c61f73f46def3305ca11d038c46665b65

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, unknown): “CPO [83] uses sequence likelihood as a reward and trains alongside an SFT objective.”
Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive (2024, background): “CPO Very recently, concurrent work [Xu et al., 2024] proposes adding a new term to the DPO loss function in order to allow DPO to become better at rejecting ‘worse’ completions that are good quality but not perfect.”
Model Alignment as Prospect Theoretic Optimization (2024, background): “Other preference-based losses include CPO (Xu et al., 2024) and IPO (Azar et al., 2024).”
ORPO: Monolithic Preference Optimization without Reference Model (2024, background): “Azar et al. (2023) prevented potential overfitting problems in DPO through identity preference optimization (IPO).”
Evaluating Alignment Techniques for Enhancing LLM Performance in a Closed-Domain Application: A RAG Bench-Marking Study (2024, unknown): “…requirements [3] • Human-Aware Loss Functions (HALOs) encompass techniques like Contrastive Preference Optimization (CPO), Direct Preference Optimization (DPO), and Kahneman-Tversky Optimization (KTO), which enhance model training by aligning with human preferences and biases [4], [5], [6].”
Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Models (2026, unknown): “To show the flexibility of the DKPS framework, we also consider applying the DKPS machinery in the context of Contrastive Preference Optimization (CPO) [39], a leading approach for the post-training of a model.”