Evaluating the Morphosyntactic Well-formedness of Generated Texts
By Adithya Pratapa, Antonios Anastasopoulos, Shruti Rijhwani, Aditi Chaudhary, David R. Mortensen, Graham Neubig, Yulia Tsvetkov. 2021
11 citations recorded by the source corpus.
Text generation systems are ubiquitous in natural language processing applications. However, evaluation of these systems remains a challenge, especially in multilingual settings. In this paper, we propose L’AMBRE – a metric to evaluate the morphosyntactic well-formedness of text using its dependency parse and morphosyntactic rules of the language. We present a way to automatically extract various rules governing morphosyntax directly from dependency treebanks. To tackle the noisy outputs from text generation systems, we propose a simple methodology to train robust parsers. We show the effectiveness of our metric on the task of machine translation through a diachronic study of systems translating into morphologically-rich languages.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Elizabeth Clark
- Sebastian Gehrmann
- Thibault Sellam
- Yejin Choi
- Percy Liang
- Noah A. Smith
- Mohit Iyyer
- Jianfeng Gao
In Natural Language Processing
- Thibault Sellam
- Elizabeth Clark
- Sebastian Gehrmann
- Yejin Choi
- Yonatan Belinkov
- Tatsunori B. Hashimoto
- Greg Durrett
- Percy Liang
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “, 2005),5 automatically constructed content units (Tauchmann & Mieskes, 2020) using the Pyramid framework by Nenkova and Passonneau (2004), dependency parses (Pratapa et al., 2021), semantic parses (Ribeiro et al.”