Resource Interoperability for Sustainable Benchmarking: The Case of Events

By C. V. Son, Oana Inel, R. Morante, Lora Aroyo, Piek T. J. M. Vossen. 2018

7 citations recorded by the source corpus.

With the continuous growth of benchmark corpora, which often annotate the same documents, there is a range of opportunities to compare and combine similar and complementary annotations. However, these opportunities are hampered by a wide range of problems that are related to the lack of resource interoperability. In this paper, we illustrate these problems by assessing aspects of interoperability at the document-level across a set of 20 corpora annotated with (aspects of) events. The issues range from applying different document naming conventions, to mismatches in textual content and structural/conceptual differences among annotation schemes. We provide insight into the exact document intersections between the corpora by mapping their document identifiers and perform an empirical analysis of event annotations showing their compatibility and consistency in and across the corpora. This way, we aim to make the community more aware of the challenges and opportunities and to inspire working collaboratively towards interoperable resources.

Subfields: Artificial Intelligence, Marketing, Molecular Biology, Evaluation & Benchmarks, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Lora Aroyo
  2. Chris Welty
  3. Praveen K. Paritosh
  4. Nicola Guarino
  5. Martin Hepp
  6. Ana Marasovic
  7. Sarah Wiegreffe
  8. Gianluca Demartini

In Molecular Biology

  1. Chris Welty
  2. Nicola Guarino
  3. Martin Hepp

In Evaluation & Benchmarks

  1. Chris Welty
  2. Lora Aroyo
  3. Gianluca Demartini

Citations in the corpus