Analysing Data-To-Text Generation Benchmarks

By Laura Perez-Beltrachini, Claire Gardent. 2017

26 citations recorded by the source corpus.

A generation system can only be as good as the data it is trained on. In this short paper, we propose a methodology for analysing data-to-text corpora used for training Natural Language Generation (NLG) systems. We apply this methodology to three existing benchmarks. We conclude by eliciting a set of criteria for the creation of a data-to-text benchmark which could help better support the development, evaluation and comparison of linguistically sophisticated data-to-text generators.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “For example, we may improve the training sets, make the test sets more challenging, or probe for specific skills a model must exhibit with testonly datasets (Perez-Beltrachini and Gardent, 2017; Linzen, 2020; Ribeiro et al., 2020; Schlegel et al., 2020).”
The WebNLG Challenge: Generating Text from RDF Data (2017, methodology): “1 Following (Perez-Beltrachini and Gardent, 2017), we use (Lu, 2008)’s system to compute the CTTR (Carroll, 1964).”
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets (2021, methodology): “To characterize sentence complexity, we follow [42] and use the 8-level developmental level scale proposed by [43], using the implementation provided by [44] (see scale in Appendix B).”
A Simple Recipe towards Reducing Hallucination in Neural Surface Realisation (2019, background): “Current datasets used for training neural NLG systems often include instances that do not contain the same amount of information from the input structure and the output text (Perez-Beltrachini and Gardent, 2017).”
Controlling hallucinations at word level in data-to-text generation (2021, unknown): “They were not constructed by human annotators, but rather created from Internet sources, and consist of loosely aligned table-reference pairs: in Wik-iBio, almost two thirds of the training instances contain divergences [7], and no instance has a 1-to-1 source-target alignment [46].”