Matching the Blanks: Distributional Similarity for Relation Learning

By Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, Tom Kwiatkowski. 2019

866 citations recorded by the source corpus.

General purpose relation extractors, which can model arbitrary relations, are a core aspiration in information extraction. Efforts have been made to build general purpose extractors that represent relations with their surface forms, or which jointly embed surface forms with relations from an existing knowledge graph. However, both of these approaches are limited in their ability to generalize. In this paper, we build on extensions of Harris’ distributional hypothesis to relations, as well as recent advances in learning text representations (specifically, BERT), to build task agnostic relation representations solely from entity-linked text. We show that these representations significantly outperform previous work on exemplar based relation extraction (FewRel) even without using any of that task’s training data. We also show that models initialized with our task agnostic representations, and then tuned on supervised relation extraction datasets, significantly outperform the previous methods on SemEval 2010 Task 8, KBP37, and TACRED

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

Citations in the corpus

SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, background): “BERTlarge and BERTEM+MTB from Soares et al. (2019) are the current state-of-the-art.”
How Can We Know What Language Models Know? (2019, neutral): “For example, LMs have been used to answer factoid questions (Radford et al., 2019), answer common sense queries (Trinh and Le, 2018; Sap et al., 2019), or extract factual knowledge about relations between entities (Petroni et al., 2019; Baldini Soares et al., 2019).”
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables (2020, neutral): “BERT EM [25] 66.3 76.9 BERT EM+MTB [25] 67.”
Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation (2022, methodology): “We labeled the top-K most similar439 instances with yi and augment them to Dktrain.440 Similarity Model To train Sπ, inspired by441 Soares et al. (2019), we adopt a contrastive learn-442 ing method to fine-tune a BERTbase model on C,443 whose sentences are already tagged with entities.444 Based on…”
A Two-phase Prototypical Network Model for Incremental Few-shot Relation Classification (2020, background): “Based on this intuition, a series of few-shot RC models (Mishra et al., 2017; Gao et al., 2019a; Soares et al., 2019; Gao et al., 2020) are proposed and can effectively recognize the novel relations with only a few (e.”
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2020, methodology): “BERT+MTB (Baldini Soares et al., 2019) is a method of training relation representation without supervision from a knowledge base or human annotators by matching the blanks (MTB).”
Continual Relation Learning via Episodic Memory Activation and Reconsolidation (2020, background): “The conventional RE work, including both supervised RE models (Zelenko et al., 2003; Zhou et al., 2005; Gormley et al., 2015; Socher et al., 2012; Liu et al., 2013; Zeng et al., 2014; Nguyen and Grishman, 2015; dos Santos et al., 2015; Xu et al., 2015; Liu et al., 2015; Miwa and Bansal, 2016) and di”
Neural Data Augmentation via Example Extrapolation (2021, methodology): “Previous results have reported super-human performance on FewRel (Baldini Soares et al., 2019).”
Blockwise Self-Attention for Long Document Understanding (2019, background): “, 2019b), relation extraction (Soares et al., 2019) and text retrieval (Lee et al.”
FewRel 2.0: Towards More Challenging Few-Shot Relation Classification (2019, methodology): “Many efforts (Gao et al., 2019; Soares et al., 2019) have been devoted to the new task and some of the methods even exceed human performance 1 on FewRel.”
More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction (2020, methodology): “There are mainly two ways for handling few-shot learning: (1) Metric learning learns a semantic metric on existing data and classiﬁes queries by comparing them with training examples (Koch et al., 2015; Vinyals et al., 2016; Snell et al., 2017; Baldini Soares et al., 2019).”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, methodology): “BERT-MTB [221] learns relation representations based on BERT by performing the matchingthe-blanks task and incorporating designed objectives for relation extraction.”
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2020, background): “Here, incorporating relation learning (Bal-dini Soares et al., 2019) and learning to represent relation triples in a LM, e.g., from large, fact-aligned corpora (Elsahar et al., 2018), are exciting avenues for future work.”
Consistent Representation Learning for Continual Relation Extraction (2022, methodology): “Given a sentence x = [ w 1 , . . . , w | x | ] and a pair of entities (E1 , E2) , we follow Soares et al. (2019b) augment x with four reserved word pieces to mark the begin and end of each entity mentioned in the sentence.”