Toward Controlled Generation of Text

By Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, Eric P. Xing. 2017

1,038 citations recorded by the source corpus.

Generic generation and manipulation of text is challenging and has limited success compared to recent deep generative modeling in visual domain. This paper aims at generating plausible natural language sentences, whose attributes are dynamically controlled by learning disentangled latent representations with designated semantics. We propose a new neural generative model which combines variational auto-encoders and holistic attribute discriminators for effective imposition of semantic structures. With differentiable approximation to discrete text samples, explicit constraints on independent attribute controls, and efficient collaborative learning of generator and discriminators, our model learns highly interpretable representations from even only word annotations, and produces realistic sentences with desired attributes. Quantitative evaluation validates the accuracy of sentence and attribute generation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “Similar approaches have been used in style transfer (Hu et al., 2017; Shen et al., 2017; Romanov et al., 2019) and unsupervised machine translation (Lample et al.”
Defending Against Neural Fake News (2019, background): “, 2019), but they do not lend themselves to producing controllable generations (Hu et al., 2017).”
GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “Methods for controlling text generation can be categorized broadly into two types: training a model directly for controllable generation (Keskar et al., 2019; Ficler & Goldberg, 2017; Yu et al., 2017; Hu et al., 2017) or using a discriminator to guide generation (Holtzman et al.”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “Controlled text generation [22] aims to steer the generated texts of language models towards desired contents, styles or domains.”
Tailor: Generating and Perturbing Text with Semantic Controls (2021, background): “Most generators take simple controls like tense (Hu et al., 2017), topic (Keskar et al., 2019), or sentiment polarity (Dathathri et al., 2020), which un-derspecify desired transformations.”
Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Several works (Ficler & Goldberg, 2017; Hu et al., 2017; Rajani et al., 2019; Yu et al., 2017; Ziegler et al., 2019; Keskar et al., 2019) have explored training or fine-tuning of models for controllable text generation.”
Unsupervised Machine Translation Using Monolingual Corpora Only (2017, background): “Before that, Hu et al. (2017) trained a variational autoencoder (VAE) where the decoder input is the concatenation of an unstructured latent vector, and a structured code representing the attribute of the sentence to generate.”
Counterfactual Fairness in Text Classification through Robustness (2018, methodology): “s in word vectors to make analogous substitutions when appropriate [15]. Another direction for generating better counterfactual examples could be based on defining a generative model over text, as in [6], where it is possible to modify certain attributes of the text while holding others constant and”
Learning to Generate Reviews and Discovering Sentiment (2017, background): “Hu et al. (2017) and Dong et al. (2017) both designed conditional generative models to disentangle the content of text from various attributes like sentiment or tense.”
Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset (2018, background): “…the emotional content of a text response either through a manually speciﬁed target (Zhou and Wang, 2018; Zhou et al., 2018; Wang and Wan, 2018; Hu et al., 2017; Huang et al., 2018) or through a general term to encourage higher levels of affect (Asghar et al., 2018), with evaluations focused…”
What makes a good conversation? How controllable attributes affect human judgments (2019, background): “Controllable neural text generation Researchers have proposed several approaches to control aspects of RNN-based natural language generation such as sentiment, length, speaker style and tense (Fan et al., 2018; Ficler and Goldberg, 2017; Ghazvininejad et al., 2017; Hu et al., 2017; Kikuchi et al., 2”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, background): “, 2017), tense (Hu et al., 2017), template (Zhu et al.”
Classification Accuracy Score for Conditional Generative Models (2019, background): “[35] measures the quality of generative models of text by training a sentiment analysis classifier.”
Style Transfer from Non-Parallel Text by Cross-Alignment (2017, methodology): “Our work most closely relates to approaches that do not utilize parallel data, but instead guide sentence generation from an indirect training signal (Mueller et al., 2017; Hu et al., 2017).”
Automatically Learning Data Augmentation Policies for Dialogue Tasks (2019, neutral): “…Lemley et al., 2017) and NLP (F¨urstenau and Lapata, 2009; Sen-nrich et al., 2016; Wang and Yang, 2015; Zhang et al., 2015; Jia and Liang, 2016; Kim and Rush, 2016; Hu et al., 2017; Xu et al., 2017; Xia et al., 2017; Silfverberg et al., 2017; Kaﬂe et al., 2017; Hou et al., 2018; Wang et al., 2018).”
Learning to predict the cosmological structure formation (2018, background): “Deep learning (14) is a fast-growing branch of machine learning, where recent advances have led to models that reach and sometimes exceed human performance across diverse areas, from analysis and synthesis of images (15–17), sound (18, 19), text (20, 21), and videos (22, 23) to complex control and p”
Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “(Hu et al., 2017) proposed generating sentences whose attributes are controlled by learning disentangled latent representations with designated semantics.”
Continual Learning for Text Classification with Information Disentanglement Based Regularization (2021, background): “, 2019) and controlled text generation (Hu et al., 2017), certain information (such as content and syntax) is extracted and shared among different categories and other information (such as style and polarity)”
Out-of-Domain Detection for Natural Language Understanding in Dialog Systems (2019, background): “lable text generation [35]–[37], some of which also involve an adversarial training process [38], [39] and controls the generated content.”