Disentangled Representation Learning for Non-Parallel Text Style Transfer

By Vineet John, Lili Mou, Hareesh Bahuleyan, Olga Vechtomova. 2018

332 citations recorded by the source corpus.

This paper tackles the problem of disentangling the latent representations of style and content in language models. We propose a simple yet effective approach, which incorporates auxiliary multi-task and adversarial objectives, for style prediction and bag-of-words prediction, respectively. We show, both qualitatively and quantitatively, that the style and content are indeed disentangled in the latent space. This disentangled latent representation learning can be applied to style transfer on non-parallel corpora. We achieve high performance in terms of transfer accuracy, content preservation, and language fluency, in comparison to various previous approaches.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Politeness Transfer: A Tag and Generate Approach (2020, background): “Current style transfer techniques (Shen et al., 2017; Hu et al., 2017; Fu et al., 2018; Yang et al., 2018; John et al., 2019) try to disentangle source style from content and then combine the content with the target style to generate the sentence in the target style.”
FUDGE: Controlled Text Generation With Future Discriminators (2021, background): “However, it is difficult to isolate the desired attribute from the distribution shift between G and the finetuning dataset (Hu et al., 2017; John et al., 2018; Lazaridou et al., 2020), i.e., it is nontrivial to preserve the desirable qualities of the P (X) modeled by G.”
PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction (2020, background): “…to disentangle style from content and steer generations (e.g., Shen et al., 2017; Zhang et al., 2018; Fu et al., 2018; Yang et al., 2018; Niu and Bansal, 2018; Romanov et al., 2019; Dai et al., 2019; John et al., 2019) or backtranslation setups (Prabhumoye et al., 2018; Lample et al., 2018).”
Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learning (2020, background): “To further ensure that only f ( s ) encodes content information of the utterance, following (John et al., 2018), we add two more discriminators D and assign them to predict the bag-of-words (BoW) features of the utterance based on f ( u ) and f ( s ) , respectively.”
Multiple-Attribute Text Rewriting (2018, background): “Most solutions use an adversarial approach to learn latent representations agnostic to the style of input sentences (Fu et al., 2017; Hu et al., 2017; Shen et al., 2017; Zhang et al., 2018a; Xu et al., 2018; John et al., 2018; Zhao et al., 2018).”
Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation (2019, methodology): “Following their work, many methods (Fu et al., 2018; John et al., 2018; Zhang et al., 2018a,b) has been proposed based on standard encoder-decoder architecture.”
Text Style Transfer: A Review and Experiment Evaluation (2020, methodology): “• Word Overlap : Vineet et al. [29] argued that the cosine similarity is not a sensitive metric as the original and transferred sentences may have high cosine similarity scores even the content of the sentences are different.”
“Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer (2019, methodology): “This approach has been later encapsulated in encoder-decoder frameworks (Fu et al., 2018; John et al., 2018; Zhang et al., 2018a,b).”
The Importance of Modeling Social Factors of Language: Theory and Practice (2021, background): “Ongoing research has shown that style can be controlled independently of content(Prabhumoye et al., 2018; John et al., 2019).”
LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer (2021, background): “…can largely be divided into two categories: (1) learning a latent space with controllable attributes such as those found in Shen et al. (2017); John et al. (2019) or (2) using unsupervised generative approaches from retrieval (Li et al., 2018), tagging using style phrases (Madaan et al.,…”
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021, background): “Then, a decoder, aware of the destination attribute, generated a transferred sentence (Shen et al., 2017; Hu et al., 2017; Fu et al., 2018; Zhang et al., 2018c; Xu et al., 2018; John et al., 2019).”
Synthetic Data for Deep Learning (2019, background): “[709] and has been extended into text style transfer by disentangling style and content in the latent space [285], disentangling syntax and semantics [36], DialogWAE for dialog modeling [215], Bilingual-GAN able to generate parallel sentences in two languages [479], and other works.”
Controllable Paraphrase Generation with a Syntactic Exemplar (2019, methodology): “Some recent work used multi-task learning as a way of improving the quality or disentanglement of learned representations (Zhao et al., 2017; Goyal et al., 2017; Du et al., 2018; John et al., 2018).”
Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research (2020, unknown): “, 2018) and adversarial learning-based (John et al., 2019) techniques.”
Variational Template Machine for Data-to-Text Generation (2020, methodology): “Inspired by the work in style-transfer (Hu et al., 2017b; Shen et al., 2017; Bao et al., 2019; John et al., 2018), an auxiliary loss is introduced to embed the template information of sentences into template variable z .”
Text Detoxification using Large Pre-trained Neural Models (2021, background): “Some of them disentangle the hidden representation into the representation of content and style (John et al., 2019).”
Detect and Perturb: Neutral Rewriting of Biased and Sensitive Text via Gradient-based Decoding (2021, unknown): “Representative approaches use an encoder-decoder setup with a discriminator (e,g. style) (Romanov et al., 2018; Dai et al., 2019; John et al., 2019; Aho and Ullman, 1972; Majumder et al., 2021a,b), backtranslation (Lample et al., 2018; Prabhumoye et al., 2018; Xu et al., 2019), pretraining (Duan et…”
Deep Learning for Text Style Transfer: A Survey (2020, methodology): “To focus on content information only, John et al. (2019) exclude stopwords and style-specific words.”