NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints

By Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi. 2020

177 citations recorded by the source corpus.

Conditional text generation often requires lexical constraints, i.e., which words should or shouldn’t be included in the output text. While the dominant recipe for conditional text generation has been large-scale pretrained language models that are finetuned on the task-specific training data, such models do not learn to follow the underlying constraints reliably, even when supervised with large amounts of task-specific examples. We propose NeuroLogic Decoding, a simple yet effective algorithm that enables neural language models – supervised or not – to generate fluent text while satisfying complex lexical constraints. Our approach is powerful yet efficient. It handles any set of lexical constraints that is expressible under predicate logic, while its asymptotic runtime is equivalent to conventional beam search. Empirical results on four benchmarks show that NeuroLogic Decoding outperforms previous approaches, including algorithms that handle a subset of our constraints. Moreover, we find that unsupervised models with NeuroLogic Decoding often outperform supervised models with conventional decoding, even when the latter is based on considerably larger networks. Our results suggest the limit of large-scale neural networks for fine-grained controllable generation and the promise of inference-time algorithms.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “One branch of work adopts logical controls, in which developers directly specify sets of words that should or should not appear in the output (Lu et al., 2021b; Pascual et al., 2021).”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, methodology): “In many CBS settings, constraints are added during beam search decoding to force the model to either include or exclude a speciﬁc word or group of words in the output (Anderson et al., 2017; Hokamp and Liu, 2017; Lu et al., 2021).”
NaturalProver: Grounded Mathematical Proof Generation with Language Models (2022, background): “This paves a promising path towards fully automated mathematical proof generation in natural mathematical language.”
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics (2021, methodology): “We compare with previous constrained decoding algorithms, including DBA (Post and Vilar, 2018a), N EURO L OGIC (Lu et al., 2021) and also specialized training proposed by Dinu et al. (2019).”
Gradient-based Constrained Sampling from Language Models (2022, methodology): “Dathathri et al. (2020); Krause et al. (2020a); Yang and Klein (2021b); Liu et al. (2021b) apply this approach for soft constraints defined by classifiers and LMs whereas Lu et al. (2021b,a); Pascual et al. (2021) develop heuristic control functions for keyword based constraints.”
InFillmore: Frame-Guided Language Generation with Bidirectional Context (2021, methodology): “Lu et al. (2020) generalize beam-search based methods with an al-gorithm that supports lexical constraints in the conjunctive normal form.”
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022, methodology): “Another line of works apply symbolic methods on top of LMs to improve their consistency, spanning from a lexical constraint on sequence decoding (Lu et al., 2021a) to a symbolic world model (Nye et al., 2021b) and discrete operations (Chen et al., 2019; Cobbe et al., 2021).”
Controlled Text Generation as Continuous Optimization with Multiple Constraints (2021, methodology): “Also related to this work is NEUROLOGIC DECODING [43] which uses approximate search in the discrete space (similar to beam search) to satisfy predicate logic constraints.”
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics (2022, unknown): “COLD achieves better lexical constraint coverage than NEUROLOGIC (Lu et al., 2021) – a beam-based discrete decoding algorithm specifically designed for lexically constrained generation, while producing more coherent and higher quality text than DELOREAN (Qin et al.”
Generating Sequences by Learning to Self-Correct (2022, methodology): “On the COMMONGEN benchmark, the self-corrector paired with the NeuroLogic constrained decoding algorithm (Lu et al., 2021) achieves the best results, outperforming the more sophisticated NeuroLogic-A* decoding algorithm, while being an order of magnitude faster.”
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (2021, background): “Lu et al. (2020) introduces “NeuroLogic decoding,” which uses logical constraints on neural language models to produce generations which contain (or do not contain) required (or forbidden) keywords.”