The TUNA-REG Challenge 2009: Overview and Evaluation Results

By Albert Gatt, Anya Belz, Eric Kow. 2009

99 citations recorded by the source corpus.

The GREC Task at REG '08 required participating systems to select coreference chains to the main subject of short encyclopaedic texts collected from Wikipedia. Three teams submitted a total of 6 systems, and we additionally created four baseline systems. Systems were tested automatically using a range of existing intrinsic metrics. We also evaluated systems extrinsically by applying coreference resolution tools to the outputs and measuring the success of the tools. In addition, systems were tested in a reading/comprehension experiment involving human subjects. This report describes the GREC Task and the evaluation methods, gives brief descriptions of the participating systems, and presents the evaluation results.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

The WebNLG Challenge: Generating Text from RDF Data (2017, background): “…(NLG) challenges have focused on surface realisation (Banik et al., 2013; Belz et al., 2011), referring expression generation (Belz and Gatt, 2007; Gatt et al., 2008; Gatt et al., 2009; Belz et al., 2008; Belz et al., 2009; Belz et al., 2010) and content selection (Bouayad-Agha et al., 2013).”
Generating Expressions that Refer to Visible Objects (2013, methodology): “Previous evaluation of REG algorithms have used measurements such as Uniqueness, Minimality, Dice (Belz and Gatt, 2008), and Accuracy (Gatt et al., 2009; Reiter and Belz, 2009).”
Computational Generation of Referring Expressions: A Survey (2012, background): “Gatt, Belz, and Kow (2009) let participants listen to expressions that were produced either automatically or by human speakers, and found a strong correlation between identiﬁcation accuracy and adequacy, suggesting that more adequate references also have more correct identiﬁcations.”
An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems (2009, background): “…occurred in the recent Generation Challenges evaluations of referring expression generation, which measured the correlations between human assessments of language quality and adequacy of content with task-performance measures (referent identification time and accuracy) (Gatt, Belz, and Kow 2009).”
Comparing Rating Scales and Preference Judgements in Language Evaluation (2010, methodology): “In our third pair of evaluations, we used the sys-tem outputs from the TUNA ’09 shared-task competition (Gatt et al., 2009).”