A Review of Human Evaluation for Style Transfer

By Eleftheria Briakou, Sweta Agrawal, Ke Zhang, Joel R. Tetreault, Marine Carpuat. 2021

32 citations recorded by the source corpus.

This paper reviews and summarizes human evaluation practices described in 97 style transfer papers with respect to three main evaluation aspects: style transfer, meaning preservation, and fluency. In principle, evaluations by human raters should be the most reliable. However, in style transfer papers, we find that protocols for human evaluations are often underspecified and not standardized, which hampers the reproducibility of research in this field and progress toward better human and automatic evaluation methods.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

A Recipe for Arbitrary Text Style Transfer with Large Language Models (2021, background): “Note that the guidelines for semantic preservation are not standardized in prior literature (Briakou et al., 2021); while some evaluations are strict that the outputs cannot contain any more information than the inputs, we asked the annotators not to penalize for meaning transformations which are ne”
Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer (2021, methodology): “…through the lens of formality style transfer (FoST), which is one of the most popular style dimensions considered by past ST work (Jin et al., 2020; Briakou et al., 2021a) and for which reference outputs and human judgments are available for four languages: English, Brazilian-Portuguese, French,…”
Challenges and Strategies in Cross-Cultural NLP (2022, background): “In particular, for style transfer, human evaluation is more reliable than automatic evaluation, but still suffers from non-standard evaluation protocols (Briakou et al., 2021a,b).”