The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task: Overview and Evaluation Results (WebNLG+ 2020)

By Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, Anastasia Shimorina. 2020

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “[cz] • WebNLG 2020 (Gardent et al., 2017; Castro Ferreira et al., 2020) – Verbalize subject-predicate-object triples in one or more sentences in a grammatical and natural way.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “The WebNLG+ shared task asked participants to contribute models that can realize text in Russian and English (Ferreira et al., 2020).”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “The datasets include WebNLG 2020 (Castro Ferreira et al., 2020), Clean E2E NLG (Dušek et al., 2019) and WikiLingua (Scialom et al., 2020) from GEM generation benchmark (Gehrmann et al., 2021).”
A Comprehensive Overview of Large Language Models (2023, unknown): “Another model Flan-PaLM [25] unlocks zero-shot reasoning with CoT training.”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, methodology): “For chrF++ (Popovic, 2017), TER (Snover et al., 2006), BERTScore (Zhang et al., 2020), and BLEURT (Sellam et al., 2020) we use the repository of Castro Ferreira et al. (2020).”
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, background): “…task-oriented dialogue datasets (i.e., BAGEL (Mairesse et al., 2010), SFHOTEL (Wen et al., 2015)), we focus on the WebNLG challenge (PerezBeltrachini et al., 2016; Gardent et al., 2017; Ferreira et al., 2018, 2020) as sentence available in this challenge is more representative of progress of NLG4.”
Control Prefixes for Text Generation (2021, background): “20We also report results on WebNLG+ 2020 (Castro Ferreira et al., 2020), the second official WebNLG competition, in Appendix B.”
Scaling Data-Constrained Language Models (2023, methodology): “Two of the tasks benchmarked, WebNLG [17, 34], a generation task, and bAbI [122, 57], a reasoning task, see jumps in performance as soon as code is added, possibly due to code enabling models to learn long-range state-tracking capabilities beneﬁcial for these tasks.”
Unifying Language Learning Paradigms (2022, methodology): “We use CNN/Dailymail (Hermann et al., 2015), XSUM (Narayan et al., 2018), MultiNews (Fabbri et al., 2019), SAMSum (Gliwa et al., 2019), WebNLG (Castro Ferreira et al., 2020) (English), E2E (Duˇsek et al., 2019) and CommonGen (Lin et al., 2020) to evaluate our models.”
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model (2022, methodology): “, 2019) and WebNLG 2020 (Ferreira et al., 2020) datasets which has much shorter context than the summarization task.”
Control Prefixes for Parameter-Efficient Text Generation (2021, methodology): “We also report results on the less popular WebNLG+ 2020 (Castro Ferreira et al., 2020) dataset (GEM), the second official WebNLG competition, in Appendix D.”
Leveraging Large Pretrained Models for WebNLG 2020 (2020, background): “The ofﬁcial report (Castro-Ferreira et al., 2020) should have more comprehensive comparisons and human evaluations for all submissions.”
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets (2021, background): “Languages Communicative Goal Reference Data-to-text CommonGen CG en Expand concept words into natural language [32] Czech Restaurant CR cs Verbalize an agents response from dialog act [33] E2E E2 en Describe a restaurant from key-value attributes [34] ToTTo TO en Describe highlighted cells in a tabl”
Denoising Pre-Training and Data Augmentation Strategies for Enhanced RDF Verbalization with Transformers (2020, unknown): “Castro-Ferreira et al. (2020) report human evaluation based on different criteria: data coverage, relevance, correctness, text structure and ﬂuency.”
Deep Learning for Text Style Transfer: A Survey (2020, background): “…generation task is to generate textual descriptions from structured data such as tables (Wiseman, Shieber, and Rush 2017; Parikh et al. 2020), meaning representations (Novikova, Dusek, and Rieser 2017), or Resource Description Framework triples (Gardent et al. 2017; Ferreira et al. 2020).”