A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models

By Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, Dawei Song. 2022

359 citations recorded by the source corpus.

Controllable Text Generation (CTG) is an emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used Transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the limited level of interpretability of deep neural networks, the controllability of these methods needs to be guaranteed. To this end, controllable text generation using Transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the past 3 to 4 years, targeting different CTG tasks that require different types of controlled constraints. In this article, we present a systematic critical review on the common tasks, main approaches, and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey article to summarize the state-of-the-art CTG techniques from the perspective of Transformer-based PLMs. We hope it can help researchers and practitioners in the related fields to quickly track the academic and technological frontier, providing them with a landscape of the area and a roadmap for future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “parallel, works on controlling decoding algorithms to promote or demote specific properties in the output text have been developed (Zhang et al., 2022a).”
Steering Language Models With Activation Engineering (2023, background): “…2023; Qi et al. 2023; Brown et al. 2023); • Intervening at decoding , as with guided or trainable decoding (Gu et al. 2017; Grover et al. 2019; see Zhang et al. 2022a for an overview of controlled generation and Jin et al. 2022 for textual style transfer); • Intervening on the prompt , as with…”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “02 A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models [27] 2022 arXiv NLP SC, 34”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “Moreover, LLMs perform reasoning by a probability model, which is an indecisive process [16].”