Understanding Dataset Difficulty with V-Usable Information

By Kawin Ethayarajh, Yejin Choi, Swabha Swayamdipta. 2021

351 citations recorded by the source corpus.

Estimating the difficulty of a dataset typically involves comparing state-of-the-art models to humans; the bigger the performance gap, the harder the dataset is said to be. However, this comparison provides little understanding of how difficult each instance in a given distribution is, or what attributes make the dataset difficult for a given model. To address these questions, we frame dataset difficulty -- w.r.t. a model $\mathcal{V}$ -- as the lack of $\mathcal{V}$-$\textit{usable information}$ (Xu et al., 2019), where a lower value indicates a more difficult dataset for $\mathcal{V}$. We further introduce $\textit{pointwise $\mathcal{V}$-information}$ (PVI) for measuring the difficulty of individual instances w.r.t. a given distribution. While standard evaluation metrics typically only compare different models for the same dataset, $\mathcal{V}$-$\textit{usable information}$ and PVI also permit the converse: for a given model $\mathcal{V}$, we can compare different datasets, as well as different instances/slices of the same dataset. Furthermore, our framework allows for the interpretability of different input attributes via transformations of the input, which we use to discover annotation artefacts in widely-used NLP benchmarks.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “As reference points, we also evaluated other publicly available alternatives as baselines: SteamSHP-XL (Ethayarajh et al., 2022) based on FLAN-T5-xl, the Open Assistant reward model based on DeBERTa V3 Large (He et al.”
OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, background): “…ﬁne-tuning of language models that has recently emerged as a promising approach to making large language models more in line with human preferences by generating more desirable outputs based on explicitly collected human preference data [6, 7, 8, 9, 10, 11] and thus making them more useful.”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, methodology): “We also explored the Stanford Human Preferences dataset (Ethayarajh et al., 2022), but we found that both RLHF and RLAIF policies did not show meaningful improvements over the SFT baseline after correcting for length biases as described in Appendix J.”
Qwen Technical Report (2023, methodology): “In Table 4, we report the test pairwise accuracy of PMP and reward models on diverse human preference benchmark datasets (Bai et al., 2022b; Stiennon et al., 2020; Ethayarajh et al., 2022; Lightman et al., 2023).”
AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (2023, background): “The data for human alignment can be sourced from human-generated instructions (Ganguli et al., 2022; Ethayarajh et al., 2022), or even synthesized from other strong LLMs (Havrilla, 2023).”
UltraFeedback: Boosting Language Models with High-quality Feedback (2023, neutral): “Some datasets are small in scale (Wu et al., 2023; Wang et al., 2023c) or provide only community votes as coarse-grained preferences (Ethayarajh et al., 2022; Askell et al., 2021).”
LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (2023, neutral): “There are also human preference datasets derived from discussion websites, such as Stanford SHP (Ethayarajh et al., 2022) from Reddit and H4StackExchange (Lambert et al., 2023) from StackExchange.”
ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (2023, neutral): “Finally, the training dataset for our reward model contains a total of 749 , 702 comparison pairs, with 340 , 025 from U LTRA F EEDBACK , 198 , 556 from Stanford SHP, 92 , 858 from OpenAI Summarize, and 118 , 263 from Anthropic Helpful.”
Model Alignment as Prospect Theoretic Optimization (2024, methodology): “The data used for LLM alignment, including the datasets used in this paper (e.g., SHP, HH, OASST) contain preferences of groups that are not representative of the broader population.”
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024, methodology): “In the stage of the gating layer, we use the following preference datasets: • HelpSteer [Wang et al., 2023a] (37k pairs) • UltraFeedback [Cui et al., 2023] (340k pairs) • SHP [Ethayarajh et al., 2022] (93k pairs) • HH-RLHF [Bai et al., 2022; Ganguli et al., 2022] Preference Data Pre-processing.”
Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (2024, methodology): “RLHFlow integrates data from several well-known preference sources, including HH-RLHF (Bai et al., 2022), SHP (Ethayarajh et al., 2022), HelpSteer (Wang et al., 2023), PKU-SafeRLHF (Ji et al., 2024), UltraFeedback (Cui et al., 2023), UltraInteract (Yuan et al., 2024), Distilabel-Capybara (Daniele an”
Evaluating Large Language Models at Evaluating Instruction Following (2023, methodology): “We also evaluate SteamSHP-flan-t5-xl (Ethayarajh et al., 2022), a preference model trained to provide its preference among two outputs given an instruction.”
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, neutral): “We also experimented with the Stanford Human Preferences dataset (Ethayarajh et al., 2022), but we found that both RLHF and RLAIF policies did not show meaningful improvements over the SFT baseline after correcting for length biases, using the procedure in Appendix J.”
REV: Information-Theoretic Evaluation of Free-Text Rationales (2022, neutral): “(Xu et al., 2020; Hewitt et al., 2021; Ethayarajh et al., 2022), defined as (5)”
DataComp: In search of the next generation of multimodal datasets (2023, background): “[37] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta.”
RewardBench: Evaluating Reward Models for Language Modeling (2024, methodology): “To do this, we evaluate most of the available models such those trained as classifiers, including UltraRM (Cui et al., 2023), Starling (Zhu et al., 2023a), PairRM (Jiang et al., 2023c), SteamSHP (Ethayarajh et al., 2022), models from Re-ward rAnked FineTuning (RAFT) (Dong et al., 2023), and others.”
Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstruction (2025, unknown): “For a detailed analysis of these features, refer to Appendix L.4 for SHP and Appendix L.5 for HH-RLHF, with the top coefficients visualized in Table 11.”
Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison (2024, methodology): “In contrast, the Stanford Human Preference Dataset (SHP) dataset is gathered from pairs of (presumably human) Reddit responses [24].”
RLHF Workflow: From Reward Modeling to Online RLHF (2024, methodology): “We consider two versions of the training set: • Mix1 : HH-RLHF + SHP + UltraFeedback + Summarization (Stiennon et al., 2020).”