Distant supervision for relation extraction without labeled data

By Mike D. Mintz, Steven Bills, Rion Snow, Dan Jurafsky. 2009

3,216 citations recorded by the source corpus.

Modern models of relation extraction for tasks like ACE are based on supervised learning of relations from small hand-labeled corpora. We investigate an alternative paradigm that does not require labeled corpora, avoiding the domain dependence of ACE-style algorithms, and allowing the use of corpora of any size. Our experiments use Freebase, a large semantic database of several thousand relations, to provide distant supervision. For each pair of entities that appears in some Freebase relation, we find all sentences containing those entities in a large unlabeled corpus and extract textual features to train a relation classifier. Our algorithm combines the advantages of supervised IE (combining 400,000 noisy pattern features in a probabilistic classifier) and unsupervised IE (extracting large numbers of relations from large corpora of any domain). Our model is able to extract 10,000 instances of 102 relations at a precision of 67.6%. We also analyze feature performance, showing that syntactic parse features are particularly helpful for relations that are ambiguous or lexically distant in their expression.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Transfer Learning in Natural Language Processing (2019, background): “…(Zoph et al., 2016; Yang et al., 2017; Wi-eting et al., 2016; Conneau et al., 2017; McCann et al., 2017) before turning to distant supervision approaches which use heuristics to automatically label datasets (Mintz et al., 2009; Severyn and Moschitti, 2015; Felbo et al., 2017; Yang et al., 2017).”
Snorkel: Rapid Training Data Creation with Weak Supervision (2017, background): “• Distant supervision: Distant supervision generates training labels by heuristically aligning data points with an external knowledge base, and is one of the most popular forms of weak supervision [4, 22,32].”
Reading Wikipedia to Answer Open-Domain Questions (2017, methodology): “Following previous work on distant supervision (DS) for relation extraction (Mintz et al., 2009), we use a”
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering (2019, methodology): “, 2018a), we gather labels for paragraphs during training using distant supervision (Mintz et al., 2009).”
Training Classifiers with Natural Language Explanations (2018, methodology): “Ad-ditionally, passing supervision information from the user to the model in the form of a dataset— rather than hard rules—promotes generalization in the new model (rather than memorization), similar to distant supervision (Mintz et al., 2009).”
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020, background): “, 2008), distant supervision (Mintz et al., 2009; Joshi et al., 2017), and leveraging document structure (Hermann et al.”
Concrete Problems in AI Safety (2016, neutral): “As with several of the other directions, applying or adapting these methods to recently developed advanced RL systems could be a promising area of research.”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, methodology): “For both datasets we draw upon existing Knowledge Bases (KBs), W IKIDATA and D RUG B ANK , as ground truth, utilising distant super-vision (Mintz et al., 2009) to induce the data – similar to Hewlett et al. (2016) and Joshi et al. (2017).”
WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia (2016, methodology): “Mintz et al. (2009) apply distant supervision to extracting Freebase triples (Bollacker et al., 2008) from Wikipedia text, analogous to the relational part of W IKI R EADING .”
Position-aware Attention and Supervised Data Improve Slot Filling (2017, methodology): “…(Zelenko et al., 2003; Bunescu and Mooney, 2005), where a statistical classiﬁer is trained on an annotated dataset; second, distant supervision (Mintz et al., 2009; Surdeanu et al., 2012), where a training set is formed by projecting the relations in an existing knowledge base onto textual…”
Knowledge Graph Embedding by Translating on Hyperplanes (2014, methodology): “Most existing extracting methods (Mintz et al. 2009; Riedel, Yao, and McCallum 2010; Hoffmann et al. 2011; Surdeanu et al. 2012) distantly collect evidences from an external text corpus for a candidate fact, ignoring the capability of the knowledge graph itself to reason the new fact.”
Learning Entity and Relation Embeddings for Knowledge Graph Completion (2015, methodology): “Most exiting methods (Mintz et al. 2009; Riedel, Yao, and McCallum 2010; Hoffmann et al. 2011; Surdeanu et al. 2012) take knowledge graphs as distant supervision to automatically annotate sentences in large-scale text corpora as training instances, and then extract textual features to build relation”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “The facts can be considered positively-labeled examples, which can be used as seed labels for distant supervision [156] when generating weak labels.”
Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation (2022, methodology): “…thoroughly.117 2 Related Work118 Conventional RE methods include supervised (Ze-119 lenko et al., 2002; Liu et al., 2013; Zeng et al., 2014;120 Miwa and Bansal, 2016), semi-supervised (Chen121 et al., 2006; Sun et al., 2011; Hu et al., 2020) and122 distantly supervised methods (Mintz et al.,…”
A Two-phase Prototypical Network Model for Incremental Few-shot Relation Classification (2020, background): “Currently, conventional supervised (Zeng et al., 2014; Chen et al., 2020) or distantly supervised (Mintz et al., 2009; Zhang et al., 2019) RC models are widely used and achieve remarkable performance.”
Connecting Language and Knowledge Bases with Embedding Models for Relation Extraction (2013, methodology): “Weak supervision is also a popular option for RE: Mintz et al. (2009) used Freebase to train weakly supervised relational extractors on Wikipedia, an approach generalized by the multi-instance learning frameworks (Riedel et al., 2010; Hoffmann et al., 2011; Surdeanu et al., 2012).”
Continual Relation Learning via Episodic Memory Activation and Reconsolidation (2020, background): “…2015; dos Santos et al., 2015; Xu et al., 2015; Liu et al., 2015; Miwa and Bansal, 2016) and distantly supervised models (Bunescu and Mooney, 2007; Mintz et al., 2009; Riedel et al., 2010; Hoffmann et al., 2011; Zeng et al., 2015; Lin et al., 2016; Han et al., 2018a; Bal-dini Soares et al.,…”
Matching the Blanks: Distributional Similarity for Relation Learning (2019, background): “Unlike related work in distant supervision for information extraction (Hoffmann et al., 2011; Mintz et al., 2009), we do not use relation labels at training time.”