Reformulating Unsupervised Style Transfer as Paraphrase Generation
By Kalpesh Krishna, John Wieting, Mohit Iyyer. 2020
281 citations recorded by the source corpus.
Modern NLP defines the task of style transfer as modifying the style of a given sentence without appreciably changing its semantics, which implies that the outputs of style transfer systems should be paraphrases of their inputs. However, many existing systems purportedly designed for style transfer inherently warp the input's meaning through attribute transfer, which changes semantic properties such as sentiment. In this paper, we reformulate unsupervised style transfer as a paraphrase generation problem, and present a simple methodology based on fine-tuning pretrained language models on automatically generated paraphrase data. Despite its simplicity, our method significantly outperforms state-of-the-art style transfer systems on both human and automatic evaluations. We also survey 23 style transfer papers and discover that existing automatic metrics can be easily gamed and propose fixed variants. Finally, we pivot to a more real-world style transfer setting by collecting a large dataset of 15M sentences in 11 diverse styles, which we use for an in-depth analysis of our system.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Jianfeng Gao
- Graham Neubig
- Luke S. Zettlemoyer
- Yejin Choi
- Noah A. Smith
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Mohit Bansal
- Alan L. Yuille
In Natural Language Processing
- Graham Neubig
- Mohit Bansal
- Percy Liang
- Yulia Tsvetkov
- Yejin Choi
- Luke S. Zettlemoyer
- Jianfeng Gao
- Noah A. Smith
In Natural Language Generation
- Marzena Karpinska
- Machel Reid
- Victor Zhong
- Dong Nguyen
- Anna Wegmann
- Mohit Iyyer
- Dan Jurafsky
- Luke Melas-Kyriazi
Citations in the corpus
- Mix and Match: Learning-free Controllable Text Generationusing Energy Language Models (2022, background): “…via either training domain-conditioned neural language models (Prabhumoye et al., 2020; He et al., 2020; Lample et al., 2018; Shen et al., 2017; Krishna et al., 2020; Reif et al., 2021; Ficler and Goldberg, 2017; Khalifa et al., 2021) or finetun-ing/modifying an underlying large pre-trained…”
- The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, methodology): “Each paper was published between 2018 and 2020 at ACL, NAACL, or EMNLP, and we exclude papers that use AMT to evaluate more wellestablished generation tasks like machine translation, or summarization.3 Unlike previous surveys of evaluating generated text (Çelikyilmaz et al., 2020; van der Lee et al.”
- Generative Pretraining for Paraphrase Evaluation (2021, methodology): “[37] Kalpesh Krishna, John Wieting, and Mohit Iyyer.”
- Text Detoxification using Large Pre-trained Neural Models (2021, methodology): “It turns out that the confidence of style classifier is a better style accuracy metric than a binary classifier and the acceptability classifier works better than perplexity, confirming the criticism of perplexity as a fluency metric (Krishna et al., 2020).”
- RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, methodology): “Compared to previous TST work that trained models from scratch (Hu et al., 2017; Shen et al., 2017, etc.) or fine-tuned pre-trained LMs (Krishna et al., 2020; Liu et al., 2021e; Hu and Li, 2021), our method presents a more efficient solu-tion that learns discrete prompts for a LM without updating…”
- StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer (2021, background): “, 2020), formality (Rao and Tetreault, 2018; Liu et al., 2020; Krishna et al., 2020), writing styles (Jhamtani et al.”
- Pre-Trained Language Models for Text Generation: A Survey (2022, background): “Moreover, text style transfer aims to convert a text into another style while preserving the basic semantics of input [52], such as sentiment transfer and writing style transfer [92].”
- Controlled Text Generation as Continuous Optimization with Multiple Constraints (2021, methodology): “However, once such models are trained, they are prone to degeneration [66] and biased, non-factual outputs [16, 45] as it is difficult to control the characteristics or attributes of the generated text without architectural modifications [26, 28, 35] and fine-tuning the models on attribute-specific ”
- Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework (2021, methodology): “with shifting style, often using human annotations (Xu et al., 2012, 2016; Rao and Tetreault, 2018; Krishna et al., 2020).”
- Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense (2023, result): “5).”
- NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation (2021, methodology): “The transformation follows the models and formulations by Krishna et al. (2020).”
- Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models (2022, methodology): “We note that these dimensions correspond exactly to the criteria that appear in Equation 1 in §3.1 (Krishna et al., 2020).”
- LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer (2021, methodology): “We further evaluate our model on semantic similarity and fluency using the classifiers released by Krishna et al. (2020).”
- Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering (2022, methodology): “Specifically, we focused on a style similar to that present in social media or in dialogues (Style paraphraser (Krishna et al., 2020) with Twitter and Switchboard style), and formal or casual (Style former paraphraser5 with casual and formal style).”
- A Review of Human Evaluation for Style Transfer (2021, background): “, 2019), to use location restrictions (Krishna et al., 2020), or to perform manual checks on the collected annotations (Rao and Tetreault, 2018; Briakou et al.”