Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge

By Ondrej Dusek, Jekaterina Novikova, Verena Rieser. 2019

250 citations recorded by the source corpus.

This paper provides a comprehensive analysis of the first shared task on End-to-End Natural Language Generation (NLG) and identifies avenues for future research based on the results. This shared task aimed to assess whether recent end-to-end NLG systems can generate more complex output by learning from datasets containing higher lexical richness, syntactic complexity and diverse discourse phenomena. Introducing novel automatic and human metrics, we compare 62 systems submitted by 17 institutions, covering a wide range of approaches, including machine learning architectures -- with the majority implementing sequence-to-sequence models (seq2seq) -- as well as systems based on grammatical rules and templates. Seq2seq-based systems have demonstrated a great potential for NLG in the challenge. We find that seq2seq systems generally score high in terms of word-overlap metrics and human evaluations of naturalness -- with the winning SLUG system (Juraska et al., 2018) being seq2seq-based. However, vanilla seq2seq models often fail to correctly express a given meaning representation if they lack a strong semantic control mechanism applied during decoding. Moreover, seq2seq models can be outperformed by hand-engineered systems in terms of overall quality, as well as complexity, length and diversity of outputs. This research has influenced, inspired and motivated a number of recent studies outwith the original competition, which we also summarise as part of this paper.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…has been a long history of shared tasks in the NLG community addressing a much more diverse set of tasks starting with referring expression generation (Gatt et al., 2008), but which have also covered tasks such as summarization (Syed et al., 2019) and data-to-text generation (Dusek et al., 2020).”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “To account for this aspect, we compute multiple diversity metrics, starting with those proposed for the analysis of the results of the E2E NLG challenge (Dusek et al., 2020) and by van Miltenburg et al.”
Neural Generation for Czech: Data and Baselines (2019, methodology): “We use the suite of word-overlap-based automatic metrics from the E2E NLG Challenge (Dušek et al., 2019),11 supporting BLEU (Papineni et al.”
Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech (2023, methodology): “To evaluate and compare the quality of human-writtenNLEs andChatGPT-generated ones, we use Informativeness [6] and Clarity [3], which are commonly used metrics that reflect the human perception of generated texts.”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “This type of knowledge grounding provides additional context and constraints at encoding time, similar to other types of grounding, such as visual grounding or, in the extreme case, grounding in symbolic representations as in task-based dialog (Dušek et al., 2020).”
Survey of Hallucination in Natural Language Generation (2022, background): “Moreover, there are some general post-hoc IE-based metrics that could be applied to hallucination evaluation, such as Slot Error Rate (SER) [38, 200], Content Selection (CS), Relation Generation (RG), and Content Ordering (CO) [181, 191].”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, background): “Natural Language Generation (NLG) The natural language generation challenge (Dušek et al., 2019) is a dataset for building a response generation module for task-oriented dialogue systems.”
Evaluation of Text Generation: A Survey (2020, background): “, 2020), and dialog response generation (Liu et al., 2016; Novikova et al., 2017; Deriu et al., 2019; Dusek et al., 2019; Gao et al., 2019), to name a few.”
Language Models are General-Purpose Interfaces (2022, background): “• Struct to Text is to construct a natural language description for some structured data: Common-Gen (Lin et al., 2020), E2ENLG (Dušek et al., 2020).”
Improving Text-to-Text Pre-trained Models for the Graph-to-Text Task (2020, methodology): “Yet seq2seq systems were found to perform poorly in correctness without proper semantic control (Duˇsek et al., 2020).”
Logical Natural Language Generation from Open-Domain Tables (2020, background): “, 2018) to achieve impressive performance on the existing datasets (Chen and Mooney, 2008; Liang et al., 2009; Lebret et al., 2016; Dušek et al., 2019; Wiseman et al., 2017) since the annotated text are mostly surface-level annotation without logical inference.”
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems (2020, background): “(2019) looked at slot-error-rate in the E2E challenge (Dušek et al., 2020), which involved generating short sentences (13 words on average) which communicated 8 attributes.”
Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence (2020, methodology): “We measure it by the number of unique unigrams and trigrams over system outputs, as done in Duˇsek et al. (2020).”
Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity (2020, methodology): “In recent years, there has been growing interest in going beyond pipeline-based approaches towards end-to-end (E-to-E) methods driven by recent advancements in deep learning (Lebret et al., 2016; Novikova et al., 2017b; Castro Ferreira et al., 2019; Duˇsek et al., 2020).”
Logic2Text: High-Fidelity Natural Language Generation from Logical Forms (2020, methodology): “, 2007; Lee, 2018), response generation in task-oriented dialogue systems (Wen et al., 2015; Budzianowski et al., 2018; Dušek et al., 2019), etc.”