FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation

By Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, Maosong Sun. 2018

690 citations recorded by the source corpus.

We present a Few-Shot Relation Classification Dataset (dataset), consisting of 70, 000 sentences on 100 relations derived from Wikipedia and annotated by crowdworkers. The relation of each sentence is first recognized by distant supervision methods, and then filtered by crowdworkers. We adapt the most recent state-of-the-art few-shot learning methods for relation classification and conduct thorough evaluation of these methods. Empirical results show that even the most competitive few-shot learning models struggle on this task, especially as compared with humans. We also show that a range of different reasoning skills are needed to solve our task. These results indicate that few-shot relation classification remains an open problem and still requires further research. Our detailed analysis points multiple directions for future research.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Making Pre-trained Language Models Better Few-shot Learners (2021, background): “, 2020), where a set of unlabeled examples are given; (2) meta-learning (Yu et al., 2018; Han et al., 2018; Bansal et al., 2020a,b; Bao et al., 2020), where a set of auxiliary tasks are given; and (3) in-”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “Meta-learning in NLP. Recent works have ex- plored meta-learning methods for relation classi- fication (Han et al., 2018; Gao et al., 2019), gen- eral text classification (Dou et al., 2019; Bansal et al., 2020a,b), low-resource machine transla- tion (Gu et al., 2018), cross-lingual…”
ERNIE: Enhanced Language Representation with Informative Entities (2019, methodology): “Since FewRel is built by checking whether the sentences contain facts in Wiki-data, we drop the related facts in KGs before pre-training for fair comparison.”
Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation (2022, methodology): “…CFRL needs471 to have sufficient relations as well as data and be472 suitable for few-shot learning, we create the CFRL473 benchmark based on FewRel (Han et al., 2018b).474 FewRel is a large-scale dataset for few-shot RE, 475 which contains 80 relations with hundreds of sam- 476 ples per relation.”
Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and System (2021, neutral): “IFS-R ELATION is converted from FewRel 4 (Han et al., 2018), which is a large-scale relation classiﬁ-cation dataset.”
Curriculum-Meta Learning for Order-Robust Continual Relation Extraction (2021, background): “FewRel (Han et al. 2018) is a labelled dataset which contains 80 relations and 700 instances per relation.”
A Two-phase Prototypical Network Model for Incremental Few-shot Relation Classification (2020, background): “…classified into two categories: meta-learning based models (Santoro et al., 2016; Ravi and Larochelle, 2016; Mishra et al., 2017) and metric-learning based models (Koch et al., 2015; Snell et al., 2017; Han et al., 2018; Gao et al., 2019a; Fan et al., 2019; Gao et al., 2019b; Soares et al., 2019).”
Continual Few-Shot Learning for Text Classification (2021, methodology): “On the dataset side, Han et al. (2018) introduce a few-shot relation classification dataset.”
Sentence Embedding Alignment for Lifelong Relation Extraction (2019, methodology): “, 2015) and lifelong FewRel (Han et al., 2018) to compare our proposed methods EA-EMR, EAEMR without Selection (EA-EMR NoSel), EAEMR without Alignment (EA-EMR noAlign), and EMR with the following baselines.”
Continual Relation Learning via Episodic Memory Activation and Reconsolidation (2020, methodology): “…et al., 2015; Miwa and Bansal, 2016) and distantly supervised models (Bunescu and Mooney, 2007; Mintz et al., 2009; Riedel et al., 2010; Hoffmann et al., 2011; Zeng et al., 2015; Lin et al., 2016; Han et al., 2018a; Baldini Soares et al., 2019), focuses on extracting predefined relations from text.”
Matching the Blanks: Distributional Similarity for Relation Learning (2019, methodology): “Han et al. (2018) presented FewRel as a supervised dataset, intended to evaluate models’ ability to adapt to relations from new domains at test time.”
Neural Data Augmentation via Example Extrapolation (2021, methodology): “More generally, this issue can be a chronic problem for tasks with constantly expanding output spaces, such as relation extraction (Han et al., 2018) and entity linking (Logeswaran et al.”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, background): “• Class Transfer: FewRel [27], HuffPost [43], Amazon [29], 20News [36], and Reuters [39] take part in meta-training and meta-testing but with different classes.”
Meta-Learning Improves Lifelong Relation Extraction (2019, methodology): “The 8 relations in each cluster were obtained by clustering the averaged Glove word embeddings of the relation names in the FewRel dataset.”
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2019, background): “(2) KEPLER not only shows improvements on FewRel 1.0, but also brings promotion on FewRel 2.0, which involves a medical KG that is not included in our training sources.”
Generalizing from a Few Examples (2019, background): “Recently, a new relation classification data set called FewRel [52] is released.”
Neural Snowball for Few-Shot Relation Learning (2019, methodology): “For parameters of the encoders, we follow (Han et al. 2018) for CNN and (Devlin et al.”
FewRel 2.0: Towards More Challenging Few-Shot Relation Classification (2019, methodology): “0, we build upon the FewRel dataset (Han et al., 2018) by adding a new test set in a quite different domain, and a NOTA relation choice.”