System Building Cost vs. Output Quality in Data-to-Text Generation
35 citations recorded by the source corpus.
Data-to-text generation systems tend to be knowledge-based and manually built, which limits their reusability and makes them time and cost-intensive to create and maintain. Methods for automating (part of) the system building process exist, but do such methods risk a loss in output quality? In this paper, we investigate the cost/quality trade-off in generation system building. We compare four new data-to-text systems which were created by predominantly automatic techniques against six existing systems for the same domain which were created by predominantly manual techniques. We evaluate the ten systems using intrinsic automatic metrics and human quality ratings. We find that increasing the degree to which system building is automated does not necessarily result in a reduction in output quality. We find furthermore that standard automatic evaluation metrics underestimate the quality of handcrafted systems and over-estimate the quality of automatically created systems.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Information Systems
In Natural Language Processing
In Evaluation & Benchmarks
Citations in the corpus
- Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech (2023, methodology): “To evaluate and compare the quality of human-writtenNLEs andChatGPT-generated ones, we use Informativeness [6] and Clarity [3], which are commonly used metrics that reflect the human perception of generated texts.”
- Comparing Rating Scales and Preference Judgements in Language Evaluation (2010, methodology): “The experiment involved 22 forecast dates and outputs from the 10 systems described in (Belz and Kow, 2009) (also included in the corpus release) for those dates (as well as the corresponding forecasts in the corpus) in the evaluation, i.e. a total of 242 forecast texts.”
- Chain of Explanation: New Prompting Method to Generate Quality Natural Language Explanation for Implicit Hate Speech (2022, background): “Clarity [2] measures how clear the meaning of the NLE is expressed (e.g., 1 = Unclear and 7 = Very Clear).”
- A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021, background): “In the NLG field, Clarity implies that text is easily understood (Belz and Kow, 2009; van der Lee et al., 2017) and that the reader is familiar with basic information introduced in the text (Lampouras and Androutsopoulos, 2013).”
- Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing (2020, background): “99 ) are often reported (Belz and Kow, 2009; Gardent et al., 2017; Duˇsek et al., 2020) for pairs of apparently very different quality criteria (e.g. Read-ability/Meaning Similarity), even when assessed separately for the express purpose of avoiding con-flation (Mille et al., 2018, 2019; Duˇsek et…”