The TUNA Challenge 2008: Overview and Evaluation Results

By Albert Gatt, Anya Belz, Eric Kow. 2008

55 citations recorded by the source corpus.

The TUNA Challenge was a set of three shared tasks at REG ’08, all of which used data from the TUNA Corpus. The three tasks covered attribute selection for referring expressions ( TUNA - AS ), realisation ( TUNA - R ) and end-to-end referring expression generation ( TUNA - REG ). 8 teams submitted a total of 33 systems to the three tasks, with an additional submission to the Open Track. The evaluation used a range of automatically computed measures. In addition, an evaluation experiment was carried out using the peer outputs for the TUNA - REG task. This report describes each task and the evaluation methods used, and presents the evaluation results.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…has been a long history of shared tasks in the NLG community addressing a much more diverse set of tasks starting with referring expression generation (Gatt et al., 2008), but which have also covered tasks such as summarization (Syed et al., 2019) and data-to-text generation (Dusek et al., 2020).”
The WebNLG Challenge: Generating Text from RDF Data (2017, background): “…(NLG) challenges have focused on surface realisation (Banik et al., 2013; Belz et al., 2011), referring expression generation (Belz and Gatt, 2007; Gatt et al., 2008; Gatt et al., 2009; Belz et al., 2008; Belz et al., 2009; Belz et al., 2010) and content selection (Bouayad-Agha et al., 2013).”
Evaluation of Text Generation: A Survey (2020, background): “The masi metric (Gatt et al., 2008), on the other hand, adapts the Jaccard coefficient, which biases it in favour of similarity when a set of attributes is a subset of the other attribute set.”
The TUNA-REG Challenge 2009: Overview and Evaluation Results (2009, background): “10 In previous TUNA identiﬁcation experiments (Belz and Gatt, 2007; Gatt et al., 2008), subjects had to read the description before identifying the intended referent.”
Computational Generation of Referring Expressions: A Survey (2012, background): “9), as one might expect, but neither correlates well with Levenshtein distance (Gatt, Belz, and Kow 2008).”
An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems (2009, background): “…experiments in which participants were presented with generated referring expressions and asked to identify the target referent (Belz and Gatt 2007; Gatt, Belz, and Kow 2008, 2009); these were carried out in conjunction with shared-task events organized under the Generation Challenges initiative…”