FewRel 2.0: Towards More Challenging Few-Shot Relation Classification

By Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou. 2019

279 citations recorded by the source corpus.

We present FewRel 2.0, a more challenging task to investigate two aspects of few-shot relation classification models: (1) Can they adapt to a new domain with only a handful of instances? (2) Can they detect none-of-the-above (NOTA) relations? To construct FewRel 2.0, we build upon the FewRel dataset by adding a new test set in a quite different domain, and a NOTA relation choice. With the new dataset and extensive experimental analysis, we found (1) that the state-of-the-art few-shot relation classification models struggle on these two aspects, and (2) that the commonly-used techniques for domain adaptation and NOTA detection still cannot handle the two challenges well. Our research calls for more attention and further efforts to these two real-world issues. All details and resources about the dataset and baselines are released at https://github.com/thunlp/fewrel.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Making Pre-trained Language Models Better Few-shot Learners (2021, methodology): “, 2016), and more specifically BERT-PAIR (Gao et al., 2019), used in meta-”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “Meta-learning in NLP. Recent works have ex- plored meta-learning methods for relation classi- fication (Han et al., 2018; Gao et al., 2019), gen- eral text classification (Dou et al., 2019; Bansal et al., 2020a,b), low-resource machine transla- tion (Gu et al., 2018), cross-lingual…”
Neural Data Augmentation via Example Extrapolation (2021, methodology): “In addition to in-domain Wikipedia evaluation, we also evaluate on out-of-domain generalization with the NYT, SemEval, and PubMed evaluation sets from FewRel 2.0 (Gao et al., 2019) and report the macro average over all domains.”
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2019, methodology): “From the results, we also have two interesting observations: (1) RoBERTa shows superior results over BERT on almost every benchmark, yet PAIR (BERT) is comparable with PAIR (RoBERTa), and even better under some settings.”
More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction (2020, background): “Gao et al. (2019) propose two problems worth further investigation:”
DeepStruct: Pretraining of Language Models for Structure Prediction (2022, methodology): “…is a long-standing challenge that relates to many NLP applications such as open information extraction (Gashteovski et al., 2019), named entity recognition (Sang and Meulder, 2003; Weischedel et al., 2013), and relation classification (Zhang et al., 2017; Han et al., 2018; Gao et al., 2019).”
AdaPrompt: Adaptive Prompt-based Finetuning for Relation Extraction (2021, background): “Some studies try to solve the challenge of insufficient labeled sentences, Some recent studies (Han et al., 2018; Gao et al., 2019; Soares et al., 2019) focus on few-shot relation extraction with meta learning or metric learning, and the goal is to develop models which can be trained with only a few”
TransPrompt: Towards an Automatic Transferable Prompting Framework for Few-shot Text Classification (2021, methodology): “, 2020), relation extraction (Gao et al., 2019), question answering (Hua et al.”
Prototypical Representation Learning for Relation Extraction (2021, background): “…attention to learning relations from the text in various specific scenarios, such as zero-few-shot scenarios (Levy et al., 2017; Han et al., 2018b; Gao et al., 2019; Soares et al., 2019; Ye & Ling, 2019) and open-domain scenarios (Banko et al., 2007; Fader et al., 2011; Mausam et al., 2012; Del…”
Meta-learning for Few-shot Natural Language Processing: A Survey (2020, methodology): “4 Datasets for few-shot NLP 4.1 Class as task FewRel (Han et al., 2018 Gao et al. (2019a) and so on, and there is a leaderboard 3 .”
A Comprehensive Survey on Automatic Knowledge Graph Construction (2023, background): “[130] discuss domain adaptation for few-shot relation classiication as a game process for searching domain-invariant features.”
Structured Prediction as Translation between Augmented Natural Languages (2021, methodology): “The ﬁrst is BERT-pair (Gao et al., 2019), a sequence classiﬁcation model based on BERT, which learns to optimize the scores indicating the relation between a query instance and other supporting instances for the same relation.”