Position-aware Attention and Supervised Data Improve Slot Filling

By Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, Christopher D. Manning. 2017

979 citations recorded by the source corpus.

Organized relational knowledge in the form of “knowledge graphs” is important for many applications. However, the ability to populate knowledge bases with facts automatically extracted from documents has improved frustratingly slowly. This paper simultaneously addresses two issues that have held back prior work. We first propose an effective new model, which combines an LSTM sequence model with a form of entity position-aware attention that is better suited to relation extraction. Then we build TACRED, a large (119,474 examples) supervised relation extraction dataset obtained via crowdsourcing and targeted towards TAC KBP relations. The combination of better supervised data and a more appropriate high-capacity model enables much better relation extraction performance. When the model trained on this new dataset replaces the previous relation extraction component of the best TAC KBP 2015 slot filling system, its F1 score increases markedly from 22.2% to 26.7%.

Subfields: Computer Vision and Pattern Recognition, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Systems

In Natural Language Processing

Citations in the corpus

Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, background): “Relation extraction systems must understand when a possible relationship between two entities is not entailed by the text (Zhang et al., 2017).”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, background): “A different ERNIE (Zhang et al., 2019) focuses on integrating structured knowledge bases with contextualized representations with an eye on knowledge-driven tasks like entity typing and relation classification.”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “For the RE task, we use a position-aware attention model (Zhang et al. 2017) that consists of a word embedding, position embedding, LSTM, attention layer, linear layer, and softmax layer.”
Knowledge Enhanced Contextual Word Representations (2019, methodology): “Relation extraction Our first task is relation extraction using the TACRED (Zhang et al., 2017) and SemEval 2010 Task 8 (Hendrickx et al., 2009) datasets.”
ExpBERT: Representation Engineering with Natural Language Explanations (2020, methodology): “We exclude these results on TACRED as it was not studied in Hancock et al. (2018), so we did not have a corresponding semantic parser and set of programmatic explanations.”
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, methodology): “For example, doubling the TACRED training data from 5000 to 10000, increases the baseline accuracy by 1.28 (p=.0017).”
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables (2020, methodology): “For relation extraction we choose two datasets, TACRED [23] and SemEval [21] in our experiments.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “Here we only include TacRED (Zhang et al., 2017) dataset and create several different tasks based on it.”
ERNIE: Enhanced Language Representation with Informative Entities (2019, methodology): “Zhang et al. (2017) propose PA-LSTM introducing a position-aware attention mechanism over an LSTM network, which evaluates the relative contribution of each word in the sequence for the ﬁnal sentence representation.”
Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation (2022, methodology): “483 In addition, to demonstrate the generalizability 484 of our method, we also create a CFRL benchmark 485 based on the TACRED dataset (Zhang et al., 2017) 486 which contains only 42 relations.”
Curriculum-Meta Learning for Order-Robust Continual Relation Extraction (2021, background): “Considering the special relation “n/a” (i.e, not available) in TACRED, we follow (?) and filter out these examples with the relation “n/a” and use the remaining 13,012 examples for Continual-TACRED.”
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2020, methodology): “This demonstrates linguistic knowledge is more useful on TACRED dataset.”
Continual Relation Learning via Episodic Memory Activation and Reconsolidation (2020, neutral): “(3) TACRED (Zhang et al., 2017).”
Troubling Trends in Machine Learning Scholarship (2018, background): “In contrast, many papers perform good ablation analyses [41, 45, 77, 82], and even retrospective attempts to isolate the source of gains can lead to new discoveries [10, 65].”
FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation (2018, neutral): “, 2008), TACRED dataset (Zhang et al., 2017), and NYT-10 dataset (Riedel et al.”
Matching the Blanks: Distributional Similarity for Relation Learning (2019, methodology): “This approach is analogous to previous work where positional embeddings have been applied to relation extraction (Zhang et al., 2017; Bilan and Roth, 2018).”
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2019, methodology): “To incorporate world knowledge into our pre-trained language representation models (PLMs), we design a multi-task loss as shown in Figure 1 and Equation 1, where L KE represents knowledge embedding loss and L LM represents language model loss.”
PTR: Prompt Tuning with Rules for Text Classification (2021, methodology): “Four popular benchmarks of relation classiﬁcation are used for our experiments, including TA-CRED (Zhang et al., 2017), TACREV (Alt et al., 2020), ReTACRED (Stoica et al., 2021), and Se-mEval 2010 Task 8 (Hendrickx et al., 2010).”