Findings of the E2E NLG Challenge

By Ondrej Dusek, Jekaterina Novikova, Verena Rieser. 2018

120 citations recorded by the source corpus.

This paper summarises the experimental setup and results of the first shared task on end-to-end (E2E) natural language generation (NLG) in spoken dialogue systems. Recent end-to-end generation systems are promising since they reduce the need for data annotation. However, they are currently limited to small, delexicalised datasets. The E2E NLG shared task aims to assess whether these novel approaches can generate better-quality output by learning from a dataset containing higher lexical richness, syntactic complexity and diverse discourse phenomena. We compare 62 systems submitted by 17 institutions, covering a wide range of approaches, including machine learning architectures – with the majority implementing sequence-to-sequence models (seq2seq) – as well as systems based on grammatical rules and templates.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “For example, the E2E NLG dataset was used for a shared task with a private test set that contained completely unseen attribute combinations, leading to a drop in performance (Dušek et al., 2018).”
Semantic Noise Matters for Neural Natural Language Generation (2019, neutral): “This phenomenon is also observed by Duˇsek et al. (2018, 2019) for systems in the E2E NLG challenge, but stands in contrast to work on related tasks, which mostly reports on hallucinations (i.e. adding information not grounded in the input), as observed for image captioning (Rohrbach et al., 2018),…”
A Survey of Data Augmentation Approaches for NLP (2021, background): “Two popular recent benchmarks are E2E-NLG (Dušek et al., 2018) and WebNLG (Gardent et al., 2017).”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “…3.1), (2) automatic conversion of questions in WikiSQL to declarative sentences (Section 3.2), and (3) incorporation of existing datasets including WebNLG 2017 (Gardent et al., 2017a,b; Shi-morina and Gardent, 2018) and Cleaned E2E (Novikova et al., 2017b; Dušek et al., 2018, 2019) (Section 3.3).”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, positive): “This paper provides a comprehensive final report and extended analysis of the first shared task on End-to-End (E2E) Natural Language Generation (NLG), substantially extending previous reports (Novikova and Rieser, 2016; Novikova et al., 2017b; Dušek et al., 2018).”
Evaluation of Text Generation: A Survey (2020, background): “7 E2E NLG Challenge Introduced in 2018, E2E NLG Challenge (Dušek et al., 2018) provided a high quality and large quantity training dataset for evaluating response generation models in spoken dialog systems.”
End-to-End Content and Plan Selection for Data-to-Text Generation (2018, methodology): “We next report results of experiments on a held-out test set, conducted by the E2E NLG challenge organizers (Duˇsek et al., 2018), shown in Table 5.”
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems (2020, methodology): “Semantic control techniques for bringing the data into closer alignment with the text (Dušek et al., 2019) have been shown to be useful with less complex datasets such as the E2E Challenge (Dušek et al., 2018).”
Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding (2021, background): “More related to the task of dialogue systems, Dušek et al. (2018, 2020) demonstrate that state-of-the-art natural language generation (NLG) models suffer from the hallucination problem.”
Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence (2020, methodology): “it by the number of unique unigrams and trigrams over system outputs, as done in Dušek et al. (2020).”
Leveraging Large Pretrained Models for WebNLG 2020 (2020, methodology): “, 2002) implemented by the e2e-metrics (Dušek et al., 2018).”
Non-neural Models Matter: a Re-evaluation of Neural Referring Expression Generation Systems (2022, background): “However, if complex rule/template-based NLG systems are taken into account, Dušek et al. (2018) found that although these systems cannot defeat neural approaches, they still have competitive performance.”
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics (2021, methodology): “More-over, we ﬁnd that N EURO L OGIC achieves state-of-the-art performance in various settings, including WMT17 English-German machine translation with lexical constraints (Dinu et al., 2019) and few-shot E2ENLG table-to-text generation (Chen et al., 2020b).”
TNT-NLG , System 2 : Data Repetition and Meaning Representation Manipulation to Improve Neural Generation (2018, methodology): “Qualitative evaluation of competing systems was done on CrowdFlower by the organizers, evaluating the 19 primary systems and Dusek et al.’s baseline system (2016).”
The ReproGen Shared Task on Reproducibility of Human Evaluations in NLG: Overview and Results (2021, background): “Dušek et al. (2018): Findings of the E2E NLG Challenge: 1 evaluation study; English; MTurk; 2 quality criteria; reproduction target: primary scores.”
Spot the Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020, methodology): “To create the ranking, we follow the approach by Duˇsek et al. (2018), where the ranking is generated by the TrueSkill (Herbrich et al., 2006) algorithm based on the win rate, and significant differences in performance are determined by bootstrap sampling.”