CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling

By Ning Miao, Hao Zhou, Lili Mou, Rui Yan, Lei Li. 2018

211 citations recorded by the source corpus.

In real-world applications of natural language generation, there are often constraints on the target sentences in addition to fluency and naturalness requirements. Existing language generation techniques are usually based on recurrent neural networks (RNNs). However, it is non-trivial to impose constraints on RNNs while maintaining generation quality, since RNNs generate sentences sequentially (or with beam search) from the first word to the last. In this paper, we propose CGMH, a novel approach using Metropolis-Hastings sampling for constrained sentence generation. CGMH allows complicated constraints such as the occurrence of multiple keywords in the target sentences, which cannot be handled in traditional RNN-based approaches. Moreover, CGMH works in the inference stage, and does not require parallel corpora for training. We evaluate our method on a variety of tasks, including keywords-to-sentence generation, unsupervised sentence paraphrasing, and unsupervised sentence error correction. CGMH achieves high performance compared with previous supervised methods for sentence generation. Our code is released at https://github.com/NingMiao/CGMH

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Generative Models

Citations in the corpus

Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “…such as rejection sampling to sample from simple distributions like pGDC bin (Roller et al., 2021; Ziegler et al., 2022), and Quasi Rejection Sampling (QRS) (Eikema et al., 2022) or MCMC techniques (Miao et al., 2019; Goyal et al., 2022) to sample from more complex distributions, such as pGDC dist.”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, background): “Two related scenarios with our task is lexically constrained decoding and word ordering (Zhang and Clark, 2015; Hasler et al., 2018; Dinu et al., 2019; Hokamp and Liu, 2017; Puduppully et al., 2017; Miao et al., 2019).”
Unsupervised Paraphrase Generation using Pre-trained Language Models (2020, methodology): “While we don’t have access to exact test set used in [21], we follow the exact guideline from [21], [18] and [15] where they mention taking random 30k sample out of 140k paraphrases form QQP dataset.”
Generating Sequences by Learning to Self-Correct (2022, methodology): “…modeling Wikipedia edits (Reid & Neubig, 2022; Faltings et al., 2021; Schick et al., 2022), which relies on supervised edits, unsupervised methods (Miao et al., 2019; Liu et al., 2020) that perturb sequences with simple operations (e.g. insertion, deletion), editing with models trained on…”
Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models (2020, methodology): “CGMH trains on un-paired sentences from the Twitter training set, as outlined in the original work (Miao et al., 2019).”
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics (2022, unknown): “The recent CGMH [35] and TSMH [57], followed by [36, 14], perform constrained decoding with extended Gibbs sampling or Metropolis-Hastings sampling in the discrete text space.”
NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints (2020, methodology): “A major use case of lexical constrained decoding is paraphrase generation (Hu et al., 2019a; Kajiwara, 2019; Hu et al., 2019b; Miao et al., 2019), by negatively constraining words in the source to enforce paraphrasing.”
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning (2020, background): “Unsupervised paraphrasing requires repeating this information (Miao et al., 2019; Bao et al., 2019), as does translation, but with a bilingual transformation (Artetxe et al., 2017; Lample et al., 2018).”
A Plug-and-Play Method for Controlled Text Generation (2021, methodology): “Recent work on stochastic search (Miao et al., 2019; Sha, 2020) has approached this problem by performing bidirectional search during generation and editing the text until the constraints are fulfilled.”
Generating Fluent Adversarial Examples for Natural Languages (2019, background): “As in the work of Miao et al. (2018), MHA is limited to make proposals for at most 200 times, and we pre-select 30 candidates at each iteration.”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Some related work aimed to enforce user-specified constraints, most notably using lexical constraints (Hokamp and Liu, 2017; Hu et al., 2019; Miao et al., 2019).”
InFillmore: Frame-Guided Language Generation with Bidirectional Context (2021, methodology): “…in an editing manner: starting with a sequence of keyword constraints and ﬂeshing out 2 Fairseq-based implementation and data to be released. a sentence via editing operations such as insertion or deletion (Miao et al., 2019; Liu et al., 2019; Sha, 2020; Susanto et al., 2020; Zhang et al., 2020).”
Controlling Text Edition by Changing Answers of Specific Questions (2021, methodology): “Stochastic search tends to edit an initial sentence step-by-step, where the editing position and action can be decided by Metropolis-Hastings sampling (Miao et al., 2019), a discrete scoring function (Liu et al.”
“Nice Try, Kiddo”: Investigating Ad Hominems in Dialogue Responses (2021, unknown): “Miao et al. (2019); Zhang et al. (2020b); Susanto et al. (2020) explore insertion-based non-autoregressive decoding algorithms.”
A Controllable Model of Grounded Response Generation (2020, background): “Prior work on machine translation and language generation has sought to enforce user-speciﬁed constraints, primarily in the form of lexical constraints (Hokamp and Liu 2017; Hu et al. 2019b,a; Miao et al. 2019).”
Generating Sentences from Disentangled Syntactic and Semantic Spaces (2019, methodology): “Experiments show that DSS-VAE outperforms the traditional VAE as well as a state-of-theart Metropolis-Hastings sampling approach (Miao et al., 2019) in this task.”
NAREOR: The Narrative Reordering Problem (2021, methodology): “Miao et al. (2019) use Metropolis-Hastings sampling to determine Levenshtein edits per generation step.”
Analyzing Leakage of Personally Identifiable Information in Language Models (2023, background): “It is an instance of constrained beam search [40], in which one searches for a sentence containing a piece of PII within the specified context that maximizes some constraint (i.”
Learning to Model Editing Processes (2022, methodology): “Editing-based lexically-constrained generation has been explored by (Miao et al., 2019) and Li et al. (2020) propose a search-based method for improving conditional text generation by way of heuristics — an approach similar in spirit to editing, however often directed towards a specific task (which…”