Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer

By Eleftheria Briakou, Sweta Agrawal, Joel R. Tetreault, Marine Carpuat. 2021

39 citations recorded by the source corpus.

While the field of style transfer (ST) has been growing rapidly, it has been hampered by a lack of standardized practices for automatic evaluation. In this paper, we evaluate leading automatic metrics on the oft-researched task of formality style transfer. Unlike previous evaluations, which focus solely on English, we expand our focus to Brazilian-Portuguese, French, and Italian, making this work the first multilingual evaluation of metrics in ST. We outline best practices for automatic evaluation in (formality) style transfer and identify several models that correlate well with human judgments and are robust across languages. We hope that this work will help accelerate development in ST, where human evaluation is often challenging to collect.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sebastian Gehrmann
2. Elizabeth Clark
3. Thibault Sellam
4. Dan Jurafsky
5. Mohit Bansal
6. Anders Søgaard
7. Noah A. Smith
8. Mirac Suzgun

In Computer Vision and Pattern Recognition

1. Elizabeth Clark
2. Thibault Sellam
3. Phillip Rust
4. Anders Søgaard
5. Dan Jurafsky
6. Luke Melas-Kyriazi
7. Sebastian Gehrmann
8. Stella Frank

In Evaluation & Benchmarks

1. Sebastian Gehrmann
2. Elizabeth Clark
3. Thibault Sellam
4. Dan Jurafsky
5. Mirac Suzgun
6. Luke Melas-Kyriazi
7. Anders Søgaard
8. Stella Frank

Citations in the corpus

• Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “A notable exception is a study by Briakou et al. (2021) which assesses different learned metrics for formality transfer and uses multilingual pre-trained models such as XLM-R (Conneau et al.”
• Challenges and Strategies in Cross-Cultural NLP (2022, background): “In particular, for style transfer, human evaluation is more reliable than automatic evaluation, but still suffers from non-standard evaluation protocols (Briakou et al., 2021a,b).”
• Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models (2022, background): “content/meaning preservation (textual similarity), style transfer strength, and fluency (Mir et al., 2019; Briakou et al., 2021).”