CoCon: A Self-Supervised Approach for Controlled Text Generation

By Alvin Chan, Yew-Soon Ong, B. Pung, Aston Zhang, Jie Fu. 2020

97 citations recorded by the source corpus.

Pretrained Transformer-based language models (LMs) display remarkable natural language generation capabilities. With their immense potential, controlling text generation of such LMs is getting attention. While there are studies that seek to control high-level attributes (such as sentiment and topic) of generated text, there is still a lack of more precise control over its content at the word- and phrase-level. Here, we propose Content-Conditioner (CoCon) to control an LM's output text with a target content, at a fine-grained level. In our self-supervised approach, the CoCon block learns to help the LM complete a partially-observed text sequence by conditioning with content inputs that are withheld from the LM. Through experiments, we show that CoCon can naturally incorporate target content into generated texts and control high-level text attributes in a zero-shot manner.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “Along similar lines, Keskar et al. (2019); Gururangan et al. (2020); Chan et al. (2021) adopt a setup of training class-conditional LMs where each LM accepts "control codes" as an additional input.”
GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “…for controlling text generation can be categorized broadly into two categories: training or finetuning a model directly for controllable generation (Chan et al., 2021; Madotto et al., 2020; Keskar et al., 2019; Ziegler et al., 2019; Rajani et al., 2019; Fan et al., 2018; Ficler and Goldberg, 2017;…”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “, generating specific words or phrases (Chan et al., 2021)) or both (Khalifa et al.”
Plug-and-Play Conversational Models (2020, background): “…reinforcement learning (Ziegler et al., 2019), adversarial training (Yu et al., 2017), by pre-training models with control codes (Keskar et al., 2019; Ficler and Goldberg, 2017; Chan et al., 2020), and other various approaches (Zhang et al., 2020b; Sheng et al., 2020; Carbone and Sarti, 2020).”
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022, unknown): “Discriminator Score (DisScore) (Kannan and Vinyals, 2017; Chan et al., 2021): This method trains a discriminator with different objectives.”
Gradient-based Constrained Sampling from Language Models (2022, background): “∇ E + 2 η ( t − 1) βz ( t 1) ) ; Update et al., 2019) or fine-tuning (Gururangan et al., 2020; Chan et al., 2021) for prompt based generation and generative models for tasks such for style transfer (Lample et al., 2019; Ziegler et al., 2020; Prabhumoye et al., 2018; Yu et al., 2017).”
Controlled Text Generation as Continuous Optimization with Multiple Constraints (2021, background): “The first focuses on directly training attribute-conditional models either through fine-tuning pretrained models with attribute-specific corpora [13, 29, 5] or via training conditional generative networks [33, 76, 49, 73].”
TextSETTR: Label-Free Text Style Extraction and Tunable Targeted Restyling (2021, methodology): “, 2020), or label-free, as in CoCon (Chan et al., 2020).”
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, methodology): “…Instruction Tuning [94] Refact/Retrain computationally expensive training; higher text quality; better controllability Retrain [58],[163] Refact [14, 45, 143, 157, 163, 167] Post-process efficient training and inefficient inference; lower text quality; better controllability Guided Strategy…”
Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning (2022, background): “, specific words or phrases (Chan et al., 2021)) or both (Khalifa et al.”