Underreporting of errors in NLG output, and what to do about it

By Emiel van Miltenburg, Miruna Clinciu, Ondrej Dusek, Dimitra Gkatzia, Stephanie Inglis, L. Leppanen, Saad Mahamood, Emma Manning, S. Schoch, Craig Thomson and 1 more. 2021

42 citations recorded by the source corpus.

We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by ‘state-of-the-art’ research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Evaluation reports should further include improved error analyses, following suggestions by van Miltenburg et al. (2021) and Bender and Koller (2020) who argue for more focus on limitations in addition to aggregated scores.”
AI and the Everything in the Whole Wide World Benchmark (2021, methodology): “• System output analysis / behavioural testing (see Appendix A.2), including error analysis [e.g. van Miltenburg et al., 2021], disaggregated analysis [e.g. Raji and Buolamwini, 2019], and counterfactual analysis [e.g. Garg et al., 2019, Hutchinson et al., 2020].”