UltraFeedback: Boosting Language Models with High-quality Feedback

By Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, Maosong Sun. arXiv.org, 2023

591 citations recorded by the source corpus.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/4c8cc2383cec93bd9ea0758692f01b98a035215b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, unknown): “For training data, we generate up to 5 responses per prompt from the UltraFeedback dataset [23] and use the ArmoRM model [79] to annotate preferences between responses.”
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, methodology): “This data, the Ultra-Feedback Binarized dataset (Cui et al., 2023) 2 , comprises both chosen and rejected completions evaluated by GPT-4.”
OLMo: Accelerating the Science of Language Models (2024, background): “• Maximum sequence length: 2048 • Data: A modified form of UltraFeedback (Cui et al., 2023), with TruthfulQA prompts removed.”
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, methodology): “…compared to prior public datasets that incorporate both SFT and preference optimization (e.g., direct preference optimization with UltraFeedback [13]), but it also delivers the best results when evaluated against six baseline instruction datasets and four preference tuning methods (DPO [44],…”
A Survey on Knowledge Distillation of Large Language Models (2024, background): “These adaptations extend across a spectrum, ranging from refining the precision of medical diagnoses (Wang et al., 2023k) and providing personalized treatment recommendations (Zhu et al., 2023) to automating routine administrative processes within healthcare settings.”
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024, methodology): “Popular pairwise ranking datasets such as HH-RLHF (Bai et al., 2022) or Ultra Feedback (Cui et al., 2023) do not include an evaluation criterion e and a verbal feedback v r m ,r n .”
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, unknown): “We also include a decontaminated subset of UltraFeedback [Cui et al., 2023], which is a composition of several datasets (FalseQA [Hu et al., 2023], UltraChat [Ding et al., 2023], Evol-Instruct [Xu et al., 2023], FLAN v2 [Longpre et al., 2023]) and has shown strong performance for general preference…”
Zephyr: Direct Distillation of LM Alignment (2023, methodology): “…mix of open and proprietary models, and have previously been shown to produce strong chat models like the UltraLM (Ding et al., 2023): • UltraChat (Ding et al., 2023) is a self-refinement dataset consisting of 1.47M multi-turn dialogues generated by GPT -3.5-TURBO over 30 topics and 20 different…”
Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (2024, methodology): “Other widely used preference datasets, such as Preference 700K (Dong et al., 2024) and Nectar (Zhu et al., 2023), are similarly affected.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…STAR (en) [259], General QA EgoSchema (en) [175], ScanQA (en) [10], LSMDC (en) [201] GUI GUI-World (en) [24] Type: Text Datasets UltraFeedback (en) [48], UltraChat (en) [58], Unnatural-Instructions (en) [90], NoRobots (en) [196], MOSS (en) [221], LIMA (en) [314], SlimOrca (en) [142],…”
RLHF Workflow: From Reward Modeling to Online RLHF (2024, methodology): “We consider two versions of the training set: • Mix1 : HH-RLHF + SHP + UltraFeedback + Summarization (Stiennon et al., 2020).”
Binary Classifier Optimization for Large Language Model Alignment (2024, unknown): “Dataset We utilize three publicly available preference datasets: UltraFeedback 3 (Cui et al., 2023), Capybara 4 (Daniele and Suphavadeeprasit, 2023), and HelpSteer2 (Wang et al., 2024).”