“Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer

By Akhilesh Sudhakar, Bhargav Upadhyay, Arjun Maheswaran. 2019

182 citations recorded by the source corpus.

Text style transfer is the task of transferring the style of text having certain stylistic attributes, while preserving non-stylistic or content information. In this work we introduce the Generative Style Transformer (GST) - a new approach to rewriting sentences to a target style in the absence of parallel style corpora. GST leverages the power of both, large unsupervised pre-trained language models as well as the Transformer. GST is a part of a larger ‘Delete Retrieve Generate’ framework, in which we also propose a novel method of deleting style attributes from the source sentence by exploiting the inner workings of the Transformer. Our models outperform state-of-art systems across 5 datasets on sentiment, gender and political slant transfer. We also propose the use of the GLEU metric as an automatic metric of evaluation of style transfer, which we found to compare better with human ratings than the predominantly used BLEU score.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “Our work also draws from a strong line of research into controlling the outputs of language models (Dathathri et al., 2020; Sudhakar et al., 2019; Ziegler et al.; Keskar et al., 2019, inter alia).”
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (2021, methodology): “As a second application we consider the well-studied task of controlling the polarity of text’s sentiment (e.g., Li et al., 2018; Sudhakar et al., 2019), steering towards either positive or negative sentiment.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “Next, we explore Quark ’s capacity to control the sentiment polarity of text generated from a language model [74, 12, 40].”
Politeness Transfer: A Tag and Generate Approach (2020, methodology): “We extend prior works (Li et al., 2018; Sudhakar et al., 2019) on attribute transfer by introducing a simple pipeline – tag & generate which is an interpretable two-staged approach for content preserving style transfer.”
PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction (2020, background): “In contrast, Li et al. (2018a) introduce a modular approach (later adapted to transformer models by Sudhakar et al., 2019) that relies on drop-in replacement of attribute markers followed by language correction.”
StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer (2021, methodology): “…transfer span statistical NLP methods (Hovy, 1987; Xu et al., 2012), neural generative models (Prabhumoye et al., 2018; Lample et al., 2019; He et al., 2020), and Retrieve-and-Edit approaches (Li et al., 2018; Hashimoto et al., 2018; Guu et al., 2018; Sudhakar et al., 2019; Madaan et al., 2020).”
On-the-Fly Controlled Text Generation with Experts and Anti-Experts (2021, methodology): “As a second setting, we consider the well-studied task of controlling the sentiment of text (e.g., Li et al., 2018; Sudhakar et al., 2019), steering towards either positive or negative sentiment.”
Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models (2022, background): “The first family of approaches involves identifying and replacing distinctive style-related phrases (Li et al. (2018); Sudhakar et al. (2019); Wu et al. (2019); Madaan et al. (2020); Malmi et al. (2020); Reid and Zhong (2021), inter alia).”
Controlled Text Generation as Continuous Optimization with Multiple Constraints (2021, background): “This task has been widely studied [22, 58, 29, among others] and differs from related tasks like sentiment transfer [61, 33, 34] where flipping the sentiment usually comes at the cost of changing meaning.”
TextSETTR: Label-Free Text Style Extraction and Tunable Targeted Restyling (2021, methodology): “However there is still a gap between our label-free approach and the best performing labeled model, B-GST (Sudhakar et al., 2019).”
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020, methodology): “Note that there is a domain shift for our model, since we trained our model using the splits of He et al. (2020) Other Comparisons: We tried to compare against other recent work in style transfer based on Trans-formers, such as Dai et al. (2019) and Sudhakar et al. (2019).”
Text Detoxification using Large Pre-trained Neural Models (2021, background): “Words associated with style are typically identified either based on their frequencies as in the original paper, some works use attention weights as features (Sudhakar et al., 2019).”
Detect and Perturb: Neutral Rewriting of Biased and Sensitive Text via Gradient-based Decoding (2021, unknown): “…(Romanov et al., 2018; Dai et al., 2019; John et al., 2019; Aho and Ullman, 1972; Majumder et al., 2021a,b), backtranslation (Lample et al., 2018; Prabhumoye et al., 2018; Xu et al., 2019), pretraining (Duan et al., 2020; Zhou et al., 2021), or use retrieval framework (Sudhakar et al., 2019).”
Text Style Transfer: A Review and Experiment Evaluation (2020, methodology): “The scarcity of parallel data motivated a new breed of TST algorithms that attempt to transfer text style without parallel data [9, 11, 15, 18, 20, 22, 24, 29, 40, 41, 44, 46– 48, 54, 60, 67, 70, 72, 76, 78, 81, 82, 85, 86, 88, 90, 92, 93].”
TLDR: Extreme Summarization of Scientific Documents (2020, background): “Similar to multitask learning, training on heterogenous data annotated with control codes has been shown to improve controlled generation in autoregressive language models (Keskar et al., 2019; ElSahar et al., 2020; Sudhakar et al., 2019; Li et al., 2020).”
Explaining The Efficacy of Counterfactually-Augmented Data (2020, methodology): “…set of experiments, we rely on four state of the art style transfer models representative of diﬀerent methodologies, each representative of a diﬀerent approach to automatically generate new examples with ﬂipped labels (Hu et al., 2017; Li et al., 2018; Sudhakar et al., 2019; Madaan et al., 2020).”