Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation

By Ning Dai, Jianze Liang, Xipeng Qiu, Xuanjing Huang. 2019

221 citations recorded by the source corpus.

Disentangling the content and style in the latent space is prevalent in unpaired text style transfer. However, two major issues exist in most of the current neural models. 1) It is difficult to completely strip the style information from the semantics for a sentence. 2) The recurrent neural network (RNN) based encoder and decoder, mediated by the latent representation, cannot well deal with the issue of the long-term dependency, resulting in poor preservation of non-stylistic semantic content. In this paper, we propose the Style Transformer, which makes no assumption about the latent representation of source sentence and equips the power of attention mechanism in Transformer to achieve better style transfer and better content preservation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, methodology): “Instead, as in Dai et al. (2019a), we use the distribution ˜ p t +1 (instead of a hard sample x t +1 ), and feed it forward to obtain (a biased) estimate of the next token’s embedding and then update ∆ H t .”
A Recipe for Arbitrary Text Style Transfer with Large Language Models (2021, background): “…(Li et al., 2018) 86.2 31.1 948 Unpaired RL (Xu et al., 2018) 52.2 37.2 2,750 Dual RL (Luo et al., 2019) 85.9 55.1 982 Style transformer (Dai et al., 2019) to be a little less angsty • to be about mining • to be better written • to be less diabolical • to be more absurd • to be more…”
FUDGE: Controlled Text Generation With Future Discriminators (2021, methodology): “We measure using Transformer-XL on the topic control task as well.”
PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction (2020, methodology): “Following previous work (Dai et al., 2019), we measure ﬂuency as perplexity ( PPL ) of the output sentence using a pre-trained GPT model that has not been ﬁne-tuned for this task.”
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020, methodology): “Note that there is a domain shift for our model, since we trained our model using the splits of He et al. (2020) Other Comparisons: We tried to compare against other recent work in style transfer based on Trans-formers, such as Dai et al. (2019) and Sudhakar et al. (2019).”
Mitigating Political Bias in Language Models Through Reinforced Calibration (2021, methodology): “We take the softmax output of the last hidden states as the policy π θ , because it can be viewed as the probability we choose token x t (action a t ) given the state s t = x 1: t (Dai et al. 2019a; Dathathri et al. 2019).”
StyTr2: Image Style Transfer with Transformers (2021, methodology): “As an alternative to recurrent and convolutional neural networks, transformer [51] is first proposed for machine translation tasks and has been widely used in various NLP tasks [4, 11, 15, 36, 43, 44].”
Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models (2022, methodology): “…sampled 50 examples (25 positive-to-negative and 25 negativeto-positive) and selected the corresponding outputs from DeleteAndRetrieve (Li et al., 2018), Style Transformer (Dai et al., 2019), LLM (Reif et al., 2022), and our own method “Prompt-and-Rerank”, along with the ground-truth references.”
Detect and Perturb: Neutral Rewriting of Biased and Sensitive Text via Gradient-based Decoding (2021, unknown): “Representative approaches use an encoder-decoder setup with a discriminator (e,g. style) (Romanov et al., 2018; Dai et al., 2019; John et al., 2019; Aho and Ullman, 1972; Majumder et al., 2021a,b), backtranslation (Lample et al., 2018; Prabhumoye et al., 2018; Xu et al., 2019), pretraining (Duan et…”
A Transformer-based Framework for Neutralizing and Reversing the Political Polarity of News Articles (2021, background): “LG 1 (x) = −x logx ′, where x ′ = G(x, s) (4) Some prior work choose to randomly drop or replace tokens [24] in sentences as noisy input.”
Diffusion Models already have a Semantic Latent Space (2022, background): “…generative adversarial network Goodfellow et al. (2020) address their latent space for image editing (Ling et al. (2021), H¨ark¨onen et al. (2020), Chefer et al. (2021), Shen et al. (2020), Y¨uksel et al. (2021), Patashnik et al. (2021), Gal et al. (2021), Dai et al. (2019), Xu et al. (2022)).”
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, methodology): “For the training baselines, we compare with two strong existing methods, Style Transformer (Dai et al., 2019) and DiRR (Liu et al.”
CoCon: A Self-Supervised Approach for Controlled Text Generation (2020, background): “Text style transfer is a related area that controls texts’ attributes by translating text from one style to another [3].”
Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs (2021, background): “…with an additional trigger to control the attributes, such as sentiment (Shen et al., 2017; Hu et al., 2017; Fu et al., 2018; Peng et al., 2018; Dai et al., 2019), tense (Logeswaran et al., 2018), plots (Ammanabrolu et al., 2020; Fan et al., 2019; Tambwekar et al., 2019; Yao et al., 2019;…”
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021, methodology): “We compare the output text that CAE-T5 generates with a selection of unpaired style-transfer models described in Section 2.2 (Shen et al., 2017; Li et al., 2018; Fu et al., 2018; Luo et al., 2019; Dai et al., 2019a).”
“Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer (2019, methodology): “Lastly, Dai et al. (2019) describe the use of Transformers for style transfer in an adversarial generator-discriminator setting, by adding an additional style embedding to the trans-former.”
Complementary Auxiliary Classifiers for Label-Conditional Text Generation (2020, methodology): “On the Yelp dataset, we compare with additional previous methods, including Cross-Align (Shen et al. 2017), MultiDecoder (Fu et al. 2018), DeleteAndRetrieve (Li et al. 2018), StyleTransformer (Dai et al. 2019), Back-Translation (Prabhumoye et al. 2018), and iVAE MI (Le Fang 2019).”
TextSETTR: Label-Free Text Style Extraction and Tunable Targeted Restyling (2021, background): “Typically, labels are assumed to be available for both source and target styles (Dai et al., 2019; He et al., 2020; Lample et al., 2019; Li et al., 2018; Niu et al., 2018; Prabhumoye et al., 2018; Shen et al., 2017; Sudhakar et al., 2019; Wu et al., 2019; Yang et al., 2018).”