GERBIL - Benchmarking Named Entity Recognition and Linking consistently

By Michael Röder, Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo. 2017

95 citations recorded by the source corpus.

. The ability to compare systems from the same domain is of central importance for their introduction into complex applications. In the domains of named entity recognition and entity linking, the large number of systems and their orthogonal evaluation w.r.t. measures and datasets has led to an unclear landscape regarding the abilities and weaknesses of the different approaches. We present G ERBIL —an improved platform for repeatable, storable and citable semantic annotation experiments— and its extension since being release. G ERBIL has narrowed this evaluation gap by generating concise, archivable, human-and machine-readable experiments, analytics and diagnostics. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile, ﬁne-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights into the extension, integration and use of annotation applications. In particular, G ERBIL provides comparable results to tool developers, simplifying the discovery of strengths and weaknesses of their implementations with respect to the state-of-the-art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in a machine-processable format, allowing for the efﬁcient querying and post-processing of evaluation results. Additionally, the tool diagnostics provided by G ERBIL provide insights into the areas where tools need further reﬁnement, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. Finally, we implemented additional types of experiments including entity typing. G ERBIL aims to become a focal point for the state-of-the-art, driving the research agenda of the community by presenting comparable objective evaluation results. Furthermore, we tackle the central problem of the evaluation of entity linking, i.e., we answer the question of how an evaluation algorithm can compare two URIs to each other without being bound to a speciﬁc knowledge base. Our approach to this problem opens a way to address the deprecation of URIs of existing gold standards for named entity recognition and entity linking, a feature which is currently not supported by the state-of-the-art. We derived the importance of this feature from usage and dataset requirements collected from the G ERBIL user community, which has already carried out more than 24.000 single evaluations using our framework. Through the resulting updates, G ERBIL now supports 8 tasks, 46 datasets and 20 systems.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Autoregressive Entity Retrieval (2020, methodology): “We empirically evaluate the performance of GENRE on more than 20 datasets, spanning three families of tasks: (i) entity disambiguation, using popular datasets and settings (both in and out-of– domain); (ii) end-to-end entity linking, with the GERBIL benchmarking tool (Röder et al., 2018), by using a”
CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “We use the exact same in-domain and out-of-domain datasets as well as evaluating the InKB micro-F1 on the GERBIL benchmark platform (Röder et al., 2018).”
A Survey on Deep Learning for Named Entity Recognition (2018, background): “[180] developed GERBIL, which provides researchers, end users and developers with easy-to-use interfaces for benchmarking entity annotation tools with the aim of ensuring repeatable and archiveable experiments.”
REL: An Entity Linker Standing on the Shoulders of Giants (2020, methodology): “We use the GERBIL platform [24] for evaluation, and report on micro and macro InKB F1 scores for both EL and ED.”
End-to-End Neural Entity Linking (2018, methodology): “We conducted experiments on the most important public EL datasets using the Gerbil platform (Röder et al., 2017).”