A Human Evaluation of AMR-to-English Generation Systems
By Emma Manning, Shira Wein, Nathan Schneider. 2020
22 citations recorded by the source corpus.
Most current state-of-the art systems for generating English text from Abstract Meaning Representation (AMR) have been evaluated only using automated metrics, such as BLEU, which are known to be problematic for natural language generation. In this work, we present the results of a new human evaluation which collects fluency and adequacy scores, as well as categorization of error types, for several recent AMR generation systems. We discuss the relative quality of these systems and how our results compare to those of automatic metrics, finding that while the metrics are mostly successful in ranking systems overall, collecting human judgments allows for more nuanced comparisons. We also analyze common errors made by these systems.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sean MacAvaney
- Nazli Goharian
- Tong Xiang
- Eugene Yang
- Jimmy J. Lin
- Zeynep Akkalyoncu Yilmaz
- Rodrigo Nogueira
- Sebastian Hofstätter
In Evaluation & Benchmarks
Citations in the corpus
- The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “Moreover, while human evaluations of generated texts can provide complementary insights to automatic evaluation (Manning et al., 2020), it can also lead to contradicting results since studies often omit crucial replication details and assume different definitions of the measured quantities (Howcroft”
- ToxCCIn: Toxic Content Classification with Interpretability (2021, background): “We argue that it is not sufficient to evaluate the interpretability of prediction along with other known issues in automated evaluation (Manning et al., 2020; Novikova et al., 2017).”