Adversarial Decomposition of Text Representation

By Alexey Romanov, Anna Rumshisky, Anna Rogers, David Donahue. 2018

47 citations recorded by the source corpus.

In this paper, we present a method for adversarial decomposition of text representation. This method can be used to decompose a representation of an input sentence into several independent vectors, each of them responsible for a specific aspect of the input sentence. We evaluate the proposed method on two case studies: the conversion between different social registers and diachronic language change. We show that the proposed method is capable of fine-grained controlled change of these aspects of the input sentence. It is also learning a continuous (rather than categorical) representation of the style of the sentence, which is more linguistically realistic. The model uses adversarial-motivational training and includes a special motivational loss, which acts opposite to the discriminator and encourages a better decomposition. Furthermore, we evaluate the obtained meaning embeddings on a downstream task of paraphrase detection and show that they significantly outperform the embeddings of a regular autoencoder.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “Similar approaches have been used in style transfer (Hu et al., 2017; Shen et al., 2017; Romanov et al., 2019) and unsupervised machine translation (Lample et al., 2018a), although not for ab-stractive summarization.”
Continual Learning for Text Classification with Information Disentanglement Based Regularization (2021, background): “…al., 2019; Li et al., 2020), where text hidden representations are often disentangled into sentiment (Fu et al., 2017; John et al., 2019), content (Romanov et al., 2019; Bao et al., 2019) and syntax (Bao et al., 2019) information through supervised learning from pre-deﬁned labels (John et al.,…”
PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction (2020, background): “…to disentangle style from content and steer generations (e.g., Shen et al., 2017; Zhang et al., 2018; Fu et al., 2018; Yang et al., 2018; Niu and Bansal, 2018; Romanov et al., 2019; Dai et al., 2019; John et al., 2019) or backtranslation setups (Prabhumoye et al., 2018; Lample et al., 2018).”
On learning and representing social meaning in NLP: a sociolinguistic perspective (2021, background): “Future work could potentially build on work on disentangling representations, such as work by Akama et al. (2018), Romanov et al. (2019) and recent work motivated by Two-Factor Semantics (Webson et al., 2020).”
Detect and Perturb: Neutral Rewriting of Biased and Sensitive Text via Gradient-based Decoding (2021, unknown): “Representative approaches use an encoder-decoder setup with a discriminator (e,g. style) (Romanov et al., 2018; Dai et al., 2019; John et al., 2019; Aho and Ullman, 1972; Majumder et al., 2021a,b), backtranslation (Lample et al., 2018; Prabhumoye et al., 2018; Xu et al., 2019), pretraining (Duan et…”
Deep Learning for Text Style Transfer: A Survey (2020, methodology): “We then replace the source attribute a with the target attribute a′, and the final transferred text is generated using the combination of z and a′ (John et al. 2019; Romanov et al. 2019).”
Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs (2021, background): “For example, Romanov et al. (2019) disentangle the meaning and the form of a sentence.”
Controlling Text Edition by Changing Answers of Specific Questions (2021, background): “Explicit disentanglement (Chen et al., 2016; John et al., 2019; Romanov et al., 2019; Sha and Lukasiewicz, 2021) tends to separate the latent space into more interpretable components with explicit correspondence to specific attributes.”