RAFT: A Real-World Few-Shot Text Classification Benchmark

By Neel Alex, Eli Lifland, Lewis Tunstall, A. Thakur, Pegah Maham, C. J. Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, A. Carlier and 2 more. 2021

67 citations recorded by the source corpus.

Large pre-trained language models have shown promise for few-shot learning, completing text-based tasks given only a few task-specific examples. Will models soon solve classification tasks that have so far been reserved for human research assistants? Existing benchmarks are not designed to measure progress in applied settings, and so don't directly answer this question. The RAFT benchmark (Real-world Annotated Few-shot Tasks) focuses on naturally occurring tasks and uses an evaluation setup that mirrors deployment. Baseline evaluations on RAFT reveal areas current techniques struggle with: reasoning over long texts and tasks with many classes. Human baselines show that some classification tasks are difficult for non-expert humans, reflecting that real-world value sometimes depends on domain expertise. Yet even non-expert human baseline F1 scores exceed GPT-3 by an average of 0.11. The RAFT datasets and leaderboard will track which model improvements translate into real-world benefits at https://raft.elicit.org .

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “In order to establish the performance of OPT-IML in a similar setting (and additionally, on RAFT (Alex et al., 2021)), we instruction-tune OPT 30B and 175B on our entire benchmark of 1,991 tasks, which we call OPT-IML-Max.”
Predictability and Surprise in Large Generative Models (2022, methodology): “Finally, we note that we can extend the analysis we do here to other economically valuable real-world tasks such as those in the recent Real-World Few-Shot Text-Classification (RAFT) benchmark [1].”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “Our overall recipe, which we dub “T-Few”, performs significantly better than ICL (even against 16× larger models) and outperforms humans for the first time on the real-world few-shot learning benchmark RAFT [2] while requiring dramatically less compute and allowing for mixed-task batches during infe”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, methodology): “Finally, many benchmarks have been proposed for comparing few-shot approaches in a standardized way (e.g., Mishra et al., 2021; Bragg et al., 2021; Xu et al., 2021; Ye et al., 2021; Alex et al., 2021).”
Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias (2023, background): “…dataset in Table 2, from which we can see that the involved datasets not only have high cardinality but also come with high imbalance ratio, i.e. , the ratio of the sample size of the majority class to that of the minority class, which reflects the long-tail class issue in real applications [1].”
Efficient Few-Shot Learning Without Prompts (2022, methodology): “In order to better benchmark S ET F IT , we used the RAFT benchmark (Alex et al., 2021) which is speciﬁcally designed for benchmarking few-shot methods.”