FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding

By Yanan Zheng, Jing Zhou, Yujie Qian, Ming Ding, Jian Li, Ruslan Salakhutdinov, Jie Tang, Sebastian Ruder, Zhilin Yang. 2021

31 citations recorded by the source corpus.

The few-shot natural language understanding (NLU) task has attracted much recent attention. However, prior methods have been evaluated under a disparate set of protocols, which hinders fair comparison and measuring the progress of the field. To address this issue, we introduce an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test correlation, and stability. Under this new evaluation framework, we re-evaluate several state-of-the-art few-shot methods for NLU tasks. Our framework reveals new insights: (1) both the absolute performance and relative gap of the methods were not accurately estimated in prior literature; (2) no single method dominates most tasks with consistent performance; (3) improvements of some methods diminish with a larger pretrained model; and (4) gains from different methods are often complementary and the best combined model performs close to a strong fully-supervised baseline. We open-source our toolkit, FewNLU, that implements our evaluation framework along with a number of state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jie Tang
2. Iz Beltagy
3. Xiao Liu
4. Ming Ding
5. Arman Cohan
6. Kyle Lo
7. Zhengxiao Du
8. Zhilin Yang

In Computer Vision and Pattern Recognition

1. Ming Ding
2. Jie Tang
3. Xiao Liu
4. Jyoti Prakash Sahoo
5. Yisheng Song
6. Kaixuan Ji
7. Zhilin Yang
8. Ting-Yuan Wang

In Natural Language Processing

1. Jie Tang
2. Kyle Lo
3. Iz Beltagy
4. Arman Cohan
5. Xiao Liu
6. Ming Ding
7. Zhengxiao Du
8. Zhilin Yang

In Evaluation & Benchmarks

1. Iz Beltagy
2. Arman Cohan
3. W. Tam
4. Ming Ding
5. Yicheng Fu
6. Kaixuan Ji
7. Zhilin Yang
8. Jie Tang

In Meta & Continual Learning

1. Jyoti Prakash Sahoo
2. Ming Ding
3. Yisheng Song
4. Ting-Yuan Wang
5. Subrota Kumar Mondal
6. Puyu Cai

Citations in the corpus

• GPT Understands, Too (2021, methodology): “We then select the best pattern for P-tuning using the validation strategy of FewNLU (Zheng et al., 2021).”
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks (2021, background): “, 2021), few-shot learning (Zheng et al., 2021; Xu et al., 2021a; Zhao et al., 2021; Gu et al., 2021; Zhang et al., 2021a), relation extraction (Chen et al.”
• FLEX: Unifying Evaluation for Few-Shot NLP (2021, background): “different settings, from validation sets of various sizes [10, 24, 79] to no validation set but a large set of unlabeled examples [55, 56].”
• A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “In natural language processing, FewGLUE_64_labeled [26] is a collection of natural language tasks in English, including text inference, textual entailment, sentiment analysis, and semantic similarity, whose corpus consists of individual sentences.”
• P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks (2022, background): “3.1 Lack of Universality Lester et al. (2021); Liu et al. (2021) have been proved quite effective in many NLP applications (Wang et al., 2021a,b; Chen et al., 2021; Zheng et al., 2021; Min et al., 2021), but still fall short at replacing fine-tuning due to lack of universality, as discussed below.”