Evaluating Style Transfer for Text

By Remi Mir, Bjarke Felbo, Nick Obradovich, Iyad Rahwan. 2019

123 citations recorded by the source corpus.

Research in the area of style transfer for text is currently bottlenecked by a lack of standard evaluation practices. This paper aims to alleviate this issue by experimentally identifying best practices with a Yelp sentiment dataset. We specify three aspects of interest (style transfer intensity, content preservation, and naturalness) and show how to obtain more reliable measures of them from human evaluation than in previous work. We propose a set of metrics for automated evaluation and demonstrate that they are more strongly correlated and in agreement with human judgment: direction-corrected Earth Mover’s Distance, Word Mover’s Distance on style-masked texts, and adversarial classification for the respective aspects. We also show that the three examined models exhibit tradeoffs between aspects of interest, demonstrating the importance of evaluating style transfer models at specific points of their tradeoff plots. We release software with our evaluation metrics to facilitate research.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Recipe for Arbitrary Text Style Transfer with Large Language Models (2021, methodology): “Each tuple was scored by three raters (3,600 ratings total) on the following three axes which are standard to textual style transfer (Mir et al., 2019): (1) transfer strength (the amount that the output actually matches the target style), (2) semantic”
Automatically Neutralizing Subjective Bias in Text (2019, neutral): “The weak association between BLEU and human evaluation scores is corroborated by other research (Cha-ganty, Mussman, and Liang 2018; Mir et al. 2019).”
PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction (2020, methodology): “Lastly, given the limitations of automated metrics for evaluating generations (Liu et al., 2016; Mir et al., 2019), we obtain human judgments of model performance through crowdsourcing (§4.6).”
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, result): “The average inter-rater agreement is 0.35 in terms of Fleiss’ kappa (Fleiss and Cohen, 1973), which is fair and similar to previous work (Mir et al., 2019).”
Adhering, Steering, and Queering: Treatment of Gender in Natural Language Generation (2020, background): “Closely related to steering are the style transfer techniques applied to text [40, 49, 54], if we were to consider the gender of the author as a style.”
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021, methodology): “While recent evaluation frameworks each discussed the key evaluation aspects of one NLG task (Venkatesh et al., 2018; Mir et al., 2019; Yamshchikov et al., 2020; Fabbri et al., 2021), our framework provides a uniﬁed methodology that facilitates metric design for all the three main categories of…”
Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models (2022, background): “content/meaning preservation (textual similarity), style transfer strength, and fluency (Mir et al., 2019; Briakou et al., 2021).”
“Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer (2019, background): “The widely agreed upon goals for a style transfer system are 1) Content preservation of the non-stylistic parts of the source sentence, 2) Style transfer strength of the stylistic attributes to the target style and 3) Fluency and correct grammar of the generated target sentence (Mir et al., 2019).”
Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer (2021, background): “Mir et al. (2019) discuss the inherent trade-off between ST evaluation aspects and propose that models are evaluated at specific points of their trade-off plots.”
A Review of Human Evaluation for Style Transfer (2021, background): “(Pang and Gimpel, 2019; Mir et al., 2019).”
Deep Learning for Text Style Transfer: A Survey (2020, methodology): “There has been a lot of attention to the problems of evaluation metrics of TST and potential improvements (Pang and Gimpel 2019; Tikhonov and Yamshchikov 2018; Mir et al. 2019; Fu et al. 2019; Pang 2019; Yamshchikov et al. 2021; Jafaritazehjani et al. 2020).”
“This is a Problem, Don’t You Agree?” Framing and Bias in Human Evaluation for Natural Language Generation (2020, background): “However, naming conventions and deﬁnitions are inconsistent and may exhibit signiﬁcant overlap, such as with naturalness, gram-maticality, and ﬂuency (Mir et al., 2019; Novikova et al., 2018).”
Text Style Transfer: A Review and Experiment Evaluation (2020, methodology): “Although researchers have put in great effort to ensure the quality of the human evaluation on TST tasks, the evaluation approach has proven to be very challenging as the interpretation of the text style is subjective and may vary across individuals [51, 56 ?”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, background): “Each paper was published between 2018 and 2020 at ACL, NAACL, or EMNLP, and we exclude papers that use AMT to evaluate more wellestablished generation tasks like machine translation, or summarization.3 Unlike previous surveys of evaluating generated text (Çelikyilmaz et al., 2020; van der Lee et al.”
StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer (2021, unknown): “…to automatic scores such as BLEU, perplexity, or binary classification accuracy of style transfer (Hu et al., 2017; Lample et al., 2019; He et al., 2020), other automatic metrics (Fu et al., 2018; Mir et al., 2019) and human evaluation are also commonly used (Li et al., 2018; Shen et al., 2017).”
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020, methodology): “(1) it is unbounded and (2) unnatural sentences with common words tend to have low perplexity (Mir et al., 2019; Pang, 2019).”