Automatic Construction of Evaluation Suites for Natural Language Generation Datasets

By Simon Mille, Kaustubh D. Dhole, Saad Mahamood, Laura Perez-Beltrachini, Varun Prashant Gangal, Mihir Kale, Emiel van Miltenburg, Sebastian Gehrmann. 2021

27 citations recorded by the source corpus.

Machine learning approaches applied to NLP are often evaluated by summarizing their performance in a single number, for example accuracy. Since most test sets are constructed as an i.i.d. sample from the overall data, this approach overly simplifies the complexity of language and encourages overfitting to the head of the data distribution. As such, rare language phenomena or text about underrepresented groups are not equally included in the evaluation. To encourage more in-depth model analyses, researchers have proposed the use of multiple test sets, also called challenge sets, that assess specific capabilities of a model. In this paper, we develop a framework based on this idea which is able to generate controlled perturbations and identify subsets in text-to-scalar, text-to-text, or data-to-text settings. By applying this framework to the GEM generation benchmark, we propose an evaluation suite made of 80 challenge sets, demonstrate the kinds of analyses that it enables and shed light onto the limits of current generation models.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Natural Language Generation

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “These findings has also been corroborated for XSum (Gehrmann et al., 2021; Mille et al., 2021).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Developing a suite of evaluations or augmenting existing generation benchmarks (Mille et al., 2021) with axes of risk evaluations (Ribeiro et al., 2020) will encourage development of holistic solutions, bridging discrimination/toxicity and information-related harms—two related directions in which…”
ReCode: Robustness Evaluation of Code Generation Models (2022, background): “ButterFingers transformation randomly selects characters of the docstrings and perturbs each of them to a random subset of similar characters, it is from (Dhole et al., 2021) and is also used in (Mille et al., 2021).”
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, background): “GEM further introduced evaluation suites (Mille et al., 2021; Dhole et al., 2021) that are compatible with its datasets and test various robustness and fairness aspects of models.”
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation (2021, methodology): “As shown by Mille et al. (2021), applying filters and transformations to development/evaluation data splits allows for testing the robustness of models and for identifying possible biases; on the other hand, applying transformations and filters to training data (data augmentation) allows for…”