The WebNLG Challenge: Generating Text from RDF Data

By Claire Gardent, Anastasia Shimorina, Shashi Narayan, Laura Perez-Beltrachini. 2017

452 citations recorded by the source corpus.

The WebNLG challenge consists in mapping sets of RDF triples to text. It provides a common benchmark on which to train, evaluate and compare “microplanners”, i.e. generation systems that verbalise a given content by making a range of complex interacting choices including referring expression generation, aggregation, lexicalisation, surface realisation and sentence segmentation. In this paper, we introduce the microplanning task, describe data preparation, introduce our evaluation methodology, analyse participant results and provide a brief description of the participating systems.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Finetuned Language Models Are Zero-Shot Learners (2021, background): “WebNLG (Gardent et al., 2017) • Translation is the task of translating text from one language into a different language.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “WebNLG (Gardent et al., 2017) 29.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “We evaluate on three standard neural generation datasets for the table-to-text task: E2E (Novikova et al., 2017), WebNLG (Gardent et al., 2017), and DART (Radev et al., 2020), as shown in Table 1.”
A Survey of Data Augmentation Approaches for NLP (2021, background): “Two popular recent benchmarks are E2E-NLG (Dušek et al., 2018) and WebNLG (Gardent et al., 2017).”
LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “WebNLG is another commonly used dataset for data-to-text evaluation [10].”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “We did not use PARENT (Dhingra et al., 2019) because not all of our inputs are in a tabular form (e.g., the ones from WebNLG and E2E).”
BLEURT: Learning Robust Metrics for Text Generation (2020, methodology): “We test B LEURT ’s ability to adapt to different tasks with the WebNLG 2017 Challenge Dataset (Gar-dent et al., 2017).”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, background): “Target Quality Target Source Content Selection Wikibio (Lebret et al., 2016) 583K Biographies Noisy Wikipedia Not speciﬁed Rotowire (Wiseman et al., 2017) 4.9K Basketball Noisy Rotowire Not speciﬁed WebNLG (Gardent et al., 2017b) 25.3K 15 DBPedia categories Clean Annotator Generated Fully speciﬁed E”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “…were applied in the COCO caption generation challenge (Chen et al. 2015) as well as other NLG experiments (e.g. Lebret, Grangier, and Auli 2016; Gardent et al. 2017b; Sharma et al. 2016b): BLEU (Papineni et al. 2002) is the harmonic mean of n-gram precisions of the system output with respect…”
Findings of the E2E NLG Challenge (2018, background): “2 While there are previous studies comparing a limited number of end-to-end NLG approaches (Novikova et al., 2017a; Wise-man et al., 2017; Gardent et al., 2017), this is the ﬁrst research to evaluate novel end-to-end generation at scale and using human assessment.”
Evaluation of Text Generation: A Survey (2020, methodology): “6 WebNLG The WebNLG challenge introduced a text generation task from RDF triples to natural language text, providing a corpus and common benchmark for comparing the microplanning capacity of the generation systems that deal with resolving and using referring expressions, aggregations, lexicalization”
Few-Shot NLG with Pre-Trained Language Model (2019, methodology): “Such data-driven approaches achieve good performance on several benchmarks like E2E challenge (Novikova, Dusek, and Rieser 2017), WebNLG challenge (Gardent et al. 2017) and W IKI B IO (Lebret, Grangier, and Auli 2016).”
DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2022, methodology): “In this experiment, we evaluate the performance of our model on different natural language generation (NLG) tasks such as the E2E NLG Challenge (Novikova et al., 2017), DART (Nan et al., 2020) and WebNLG (Gardent et al., 2017).”
Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing (2020, background): “99 ) are often reported (Belz and Kow, 2009; Gardent et al., 2017; Duˇsek et al., 2020) for pairs of apparently very different quality criteria (e.g. Read-ability/Meaning Similarity), even when assessed separately for the express purpose of avoiding con-ﬂation (Mille et al., 2018, 2019; Duˇsek et…”
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders (2021, unknown): “The benchmark data is WebNLG (Gardent et al., 2017).”
The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task: Overview and Evaluation Results (WebNLG+ 2020) (2020, unknown): “Following the tradition of several test data types, introduced in the previous shared task (Gardent et al., 2017b), we kept them in this year edition and introduced one new type unseen entities.”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, background): “In recent years, researchers made signiﬁcant progresses in text generation: translation (Birch, 2021), structured data-to-text (Gardent et al., 2017), dialogue generation (Vinyals and Le, 2015), summarization (Chopra et al., 2016) and image captioning (Fang et al., 2014).”