The E2E Dataset: New Challenges For End-to-End Generation

By Jekaterina Novikova, Ondrej Dusek, Verena Rieser. 2017

530 citations recorded by the source corpus.

This paper describes the E2E data, a new dataset for training end-to-end, data-driven natural language generation systems in the restaurant domain, which is ten times bigger than existing, frequently used datasets in this area. The E2E dataset poses new challenges: (1) its human reference texts show more lexical richness and syntactic variation, including discourse phenomena; (2) generating from this set requires content selection. As such, learning from this dataset promises more natural, varied and less template-like system utterances. We also establish a baseline on this dataset, which illustrates some of the difficulties associated with this data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “[cz] • WebNLG 2020 (Gardent et al., 2017; Castro Ferreira et al., 2020) – Verbalize subject-predicate-object triples in one or more sentences in a grammatical and natural way.”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Dušek et al. (2019) find that cleaning the E2E NLG dataset (Novikova et al., 2017b), for which the communicative goal is to describe a restaurant given a set of key-value attributes, led to a reduction in slot-error rate of up to 97%, which means that failures may have incor- rectly have been…”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “…Meaning Representation DART (Radev et al., 2020) Describe cells in a table, covering all information provided in triples. en 82k Triple Set E2E clean (Novikova et al., 2017) (Dušek et al., 2019) Describe a restaurant, given all and only the attributes specified on the input. en 42k Meaning…”
Structure-to-Text Generation with Self-Training, Acceptability Classifiers and Context-Conditioning for the GEM Shared Task (2021, methodology): “In this paper, we apply these techniques to 3 of the datasets from the GEM Shared Task (Gehrmann et al., 2021): the Schema-Guided Dialog (SGD) dataset (Rastogi et al., 2019), the End-to-End (E2E) dataset (Novikova et al., 2017) and the WebNLG-en dataset (Gardent et al., 2017).”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “We evaluate on three standard neural generation datasets for the table-to-text task: E2E (Novikova et al., 2017), WebNLG (Gardent et al., 2017), and DART (Radev et al., 2020), as shown in Table 1.”
LoRA: Low-Rank Adaptation of Large Language Models (2021, methodology): “For a direct comparison with [21] on GPT-2, we follow their setup and use Table-to-Text E2E NLG Challenge [28], a language generation task.”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “We incorporate Cleaned E2E because of its strict alignment between the meaning representation and the text.”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, methodology): “The table-to-text tasks are: E2E (Novikova et al., 2017), DART (Nan et al., 2020), and ToTTo (Parikh et al., 2020).”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “• Data-to-text generation describes structured data in natural language (Novikova et al., 2017).”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, background): “Target Quality Target Source Content Selection Wikibio (Lebret et al., 2016) 583K Biographies Noisy Wikipedia Not speciﬁed Rotowire (Wiseman et al., 2017) 4.9K Basketball Noisy Rotowire Not speciﬁed WebNLG (Gardent et al., 2017b) 25.3K 15 DBPedia categories Clean Annotator Generated Fully speciﬁed E”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, background): “A recurring theme is that metrics like BLEU and NIST (Doddington, 2002) are not suitable for judging content quality in NLG. Recently, Novikova et al. (2017a) did a comprehensive study of several metrics on the outputs of state-of-the-art NLG systems, and found that while they showed acceptable…”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “System Architecture Delex. slots Copy Semantic control Data augmentation / diversity ♥TGEN (Novikova, Dušek, and Rieser 2017) seq2seq (TGEN) name, near MR classification reranking ♥ADAPT (Elder et al. 2018) seq2seq (OpenNMT-py) none X none enriching MR by output words ♥CHEN (Chen 2018) seq2seq none…”
Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “We use E2ENLG (Novikova et al., 2017) and four different domains (restaurant, hotel, tv, laptop) from RNNLG (Wen et al., 2015) to form ﬁve similar tasks.”
Lifelong Language Knowledge Distillation (2020, methodology): “Except for E2ENLG, the Seq-KD in yellow lines usually gain more performance at the end of the training stream.”
Towards a Unified View of Parameter-Efficient Transfer Learning (2021, methodology): “…the GLUE benchmark with an encoder-only model (Guo et al., 2021; Ben Zaken et al., 2021; Mahabadi et al., 2021), or on relatively simple generation benchmarks such as E2E (Novikova et al., 2017) with an encoder-decoder model (Li & Liang, 2021), may not generalize well to other standard benchmarks.”
RankME: Reliable Human Ratings for Natural Language Generation (2018, methodology): “We were able to obtain outputs of 3 systems from the recent E2E NLG challenge (Novikova et al., 2017b):1 the Sheffield NLP system (Chen et al., 2018) and the Slug2Slug system (Juraska et al., 2018), as well as the outputs of the baseline TGen system (Dušek and Jurčı́ček, 2016).”