TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task

By Christoph Alt, Aleksandra Gabryszak, Leonhard Hennig. 2020

188 citations recorded by the source corpus.

TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE). But, even with recent advances in unsupervised pre-training and knowledge enhanced neural RE, models still show a high error rate. In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? And how do crowd annotations, dataset, and models contribute to this error rate? To answer these questions, we first validate the most challenging 5K examples in the development and test sets using trained annotators. We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled. On the relabeled test set the average F1 score of a large baseline model set improves from 62.1 to 70.1. After validation, we analyze misclassifications on the challenging instances, categorize them into linguistically motivated error groups, and verify the resulting error hypotheses on three state-of-the-art RE models. We show that two groups of ambiguous relations are responsible for most of the remaining errors and that models may adopt shallow heuristics on the dataset when entities are not masked.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Entities as Experts: Sparse Memory Access with Entity Supervision (2020, background): “(2019) at knowledge probing tasks and the TACRED relation extraction task (Zhang et al., 2017; Alt et al., 2020).”
PTR: Prompt Tuning with Rules for Text Classification (2021, neutral): “TACREV: one dataset built based on the original TACRED dataset.”
Robust Retrieval Augmented Generation for Zero-shot Slot Filling (2021, methodology): “For our experiments, we adopt a revisited version of TACRED (Alt et al., 2020), in which a second stage crowdsourcing is performed to further improve the quality of the annotations and resolve conflicts among relations.”
AdaPrompt: Adaptive Prompt-based Finetuning for Relation Extraction (2021, methodology): “RE of early days has gone through pattern-based meth-ods (Huffman, 1995; Califf and Mooney, 1997), RNN-based (Zhou et al., 2016; Zhang et al., 2017) and graph-based (Zhang et al., 2018b; Guo et al., 2019) models, etc.”