The First Multilingual Surface Realisation Shared Task (SR’18): Overview and Evaluation Results

By Simon Mille, Anya Belz, Bernd Bohnet, Yvette Graham, Emily Pitler, Leo Wanner. 2018

65 citations recorded by the source corpus.

We report results from the SR’18 Shared Task, a new multilingual surface realisation task organised as part of the ACL’18 Workshop on Multilingual Surface Realisation. As in its English-only predecessor task SR’11, the shared task comprised two tracks with different levels of complexity: (a) a shallow track where the inputs were full UD structures with word order information removed and tokens lemmatised; and (b) a deep track where additionally, functional words and morphological information were removed. The shallow track was offered in ten, and the deep track in three languages. Systems were evaluated (a) automatically, using a range of intrinsic metrics, and (b) by human judges in terms of readability and meaning similarity. This report presents the evaluation results, along with descriptions of the SR’18 tracks, data and evaluation methods. For full descriptions of the participating systems, please see the separate system reports elsewhere in this volume.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “There have also been multiple shared tasks in NLG that focus on multilingualism, for instance, the shared task on multilingual surface realization which includes eleven languages (Mille et al., 2018, 2019, 2020).”
Neural Generation for Czech: Data and Baselines (2019, background): “Bohnet et al. (2010) created the ﬁrst statistical multilingual re-alizer based on a pipeline of SVMs, the recent surface realization challenge (Mille et al., 2018) then features further fully trainable realizers tested on multiple languages, including neural models.”
Neural data-to-text generation: A comparison between pipeline and end-to-end architectures (2019, unknown): “In comparison with other popular NLG benchmarks (Belz et al., 2011; Novikova et al., 2017; Mille et al., 2018), WebNLG is the most semantically varied corpus, consisting of 25,298 texts describing 9,674 sets of up to 7 RDF triples in 15 domains.”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, methodology): “Direct Assessment is also employed for evaluation of video captioning systems at TRECvid (Graham et al., 2018; Awad et al., 2019, 2021) and multilingual surface realisation (Mille et al., 2018, 2019).”
Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing (2020, background): “…reported (Belz and Kow, 2009; Gardent et al., 2017; Duˇsek et al., 2020) for pairs of apparently very different quality criteria (e.g. Read-ability/Meaning Similarity), even when assessed separately for the express purpose of avoiding con-ﬂation (Mille et al., 2018, 2019; Duˇsek et al., 2020).”
Enriching the WebNLG corpus (2018, background): “Moreover, a silver-standard version of the corpus in German is provided, hopefully making it more useful for the exploration of NLG in other languages or for the study of Multilingual Surface Realization (Mille et al., 2018).”
Enhancing AMR-to-Text Generation with Dual Graph Representations (2019, unknown): “Using the DA method inspired by Mille et al. (2018), we assess two quality criteria: (i) meaning similarity : how close in meaning the generated text is to the gold sentence; and (ii) readability : how well the generated sentence reads (Is it good ﬂuent English?).”
Findings of the 2020 Conference on Machine Translation (WMT20) (2020, methodology): “Direct Assessment is also employed for evaluation of video captioning systems at TRECvid (Graham et al., 2018; Awad et al., 2019) and multilingual surface realisation (Mille et al., 2018, 2019).”
Revisiting Challenges in Data-to-Text Generation with Fact Grounding (2020, methodology): “Various forms of structured data has been used as input for data-to-text generation tasks, such as tree (Belz et al., 2011; Mille et al., 2018), graph (Konstas and Lapata, 2012), dialog moves (Novikova et al.”
The Third Multilingual Surface Realisation Shared Task (SR’20): Overview and Evaluation Results (2020, methodology): “We adopted the same approach to human evaluation as in SR’18 (Mille et al., 2018) and SR’19 (Mille et al.”