DARE: Data Augmented Relation Extraction with GPT-2

By Yannis Papanikolaou, Andrea Pierleoni. 2020

85 citations recorded by the source corpus.

Real-world Relation Extraction (RE) tasks are challenging to deal with, either due to limited training data or class imbalance issues. In this work, we present Data Augmented Relation Extraction(DARE), a simple method to augment training data by properly fine-tuning GPT-2 to generate examples for specific relation types. The generated training data is then used in combination with the gold dataset to train a BERT-based RE classifier. In a series of experiments we show the advantages of our method, which leads in improvements of up to 11 F1 score points against a strong base-line. Also, DARE achieves new state of the art in three widely used biomedical RE datasets surpassing the previous best results by 4.7 F1 points on average.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yejin Choi
2. Mohit Bansal
3. Tie-Yan Liu
4. Tao Qin
5. Luke S. Zettlemoyer
6. Hannaneh Hajishirzi
7. Jianfeng Gao
8. Noah A. Smith

In Molecular Biology

1. Ronan Le Bras
2. Peter West
3. Tao Qin
4. Eduard H. Hovy
5. Steven Y. Feng
6. Yejin Choi
7. Hoifung Poon
8. Tie-Yan Liu

In Natural Language Processing

1. Yejin Choi
2. Mohit Bansal
3. Ronan Le Bras
4. Tie-Yan Liu
5. Noah A. Smith
6. Sean Welleck
7. Tao Qin
8. Hannaneh Hajishirzi

In Information Extraction

1. Eduard H. Hovy
2. Yejin Choi
3. Hoifung Poon
4. Tie-Yan Liu

Citations in the corpus

• Generating Datasets with Pretrained Language Models (2021, neutral): “Some approaches augment existing datasets with automatically generated examples (Anaby-Tavor et al., 2020; Papanikolaou and Pierleoni, 2020; Yang et al., 2020; Mohapatra et al., 2020; Kumar et al., 2021), but in contrast to our work, all of these approaches require that there already exists a…”
• Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, background): “With the ever-growing size and capability of pretrained language models (PLMs), some recent works explore automatically dataset synthesis and expansion by finetuning PLMs on the existing labeled data (Anaby-Tavor et al., 2020; Papanikolaou and Pierleoni, 2020; Kumar et al., 2020; Yang et al., 2020).”
• Neural Data-to-Text Generation with LM-based Text Augmentation (2021, background): “People have also also shown that GPT-2 is able to improve classification tasks via in-domain text augmentation (Papanikolaou and Pierleoni, 2020; Sun et al., 2020).”
• True Few-Shot Learning with Prompts—A Real-World Perspective (2021, unknown): “Building on earlier work (Anaby-Tavor et al., 2020; Papanikolaou and Pierleoni, 2020; Yang et al., 2020; Mohapatra et al., 2020; Kumar et al., 2020; Schick and Schütze, 2021), we investigate whether synthetic examples can replace unlabeled data.”
• Large language models encode clinical knowledge (2022, background): “LLMs for science and biomedicine Recent studies, such as SciBERT [5], BioNLP [46], BioMegatron [76], BioBERT [44], PubMedBERT [25], DARE [66], ScholarBERT [31], and BioGPT [56], have demonstrated the effectiveness of using curated scientific and biomedical corpora for both discriminative and generat”
• GenAug: Data Augmentation for Finetuning Text Generators (2020, methodology): “There has been work using GPT-2 as a component in the data augmentation process for training classifiers (Kumar et al., 2020; Papanikolaou and Pierleoni, 2020).”
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining (2022, methodology): “A previous work on pre-training GPT on biomedical literature is DARE [21].”