Pragmatically Informative Text Generation

By Sheng Shen, Daniel Fried, Jacob Andreas, Daniel Klein. 2019

71 citations recorded by the source corpus.

We improve the informativeness of models for conditional text generation using techniques from computational pragmatics. These techniques formulate language production as a game between speakers and listeners, in which a speaker should generate output text that a listener can use to correctly identify the original input that the text describes. While such approaches are widely used in cognitive science and grounded language learning, they have received less attention for more standard language generation tasks. We consider two pragmatic modeling methods for text generation: one where pragmatics is imposed by information preservation, and another where pragmatics is imposed by explicit modeling of distractors. We find that these methods improve the performance of strong existing systems for abstractive summarization and generation from structured meaning representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “GeDi most closely relates to distractor based pragmatics (Andreas and Klein, 2016; Cohn-Gordon et al., 2018; Shen et al., 2019), where a single model processes a true input and a distractor input, and uses Bayes rule to produce text that fits the true input but not the distractor input.”
DART: Open-Domain Structured Data Record to Text Generation (2020, background): “Recently, neural encoder-decoder models based on attention and copy mechanisms have shown promising results (Wiseman et al., 2017; Liu et al., 2018; Gehrmann et al., 2018; Puduppully et al., 2018, 2019; Iso et al., 2019; Castro Ferreira et al., 2019; Shen et al., 2019; Roberti et al., 2019; Zhu et a”
FUDGE: Controlled Text Generation With Future Discriminators (2021, background): “, G’s output logits), and operates directly on these logits (Ghazvininejad et al., 2017; Holtzman et al., 2018; Cohn-Gordon et al., 2018; Shen et al., 2019).”
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics (2021, methodology): “Following Shen et al. (2019), we linearize the given table into a string and ﬁnetune GPT-2 with given few-shot examples.”
Building Adaptive Acceptability Classifiers for Neural NLG (2021, methodology): “As an alternative to using acceptability classi-ﬁers, one can make use of reconstruction models (Shen et al., 2019; Yee et al., 2019) to determine how well the NLG model’s output predicts its input.”
Leveraging Large Pretrained Models for WebNLG 2020 (2020, methodology): “As shown in Shen et al. (2019); Yee et al. (2019), techniques similar to the reverse model reranking method we use here, under the guise of RSA or noisy channel models, have shown signiﬁcant beneﬁts for generation tasks when there is room for improvement.”
A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Models (2019, methodology): “Our method is in some ways similar to the re-constructor setting of Shen et al. (2019), where a base speaker model S 0 produces utterances and a listener model L reconstructs the input MR.”
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (2021, methodology): “This broad approach has been used in image generation (Ramesh et al., 2021), text generation (Holtzman et al., 2018; Shen et al., 2019; Deng et al., 2020), dialogue systems (for control, coherence and safety (Welleck et al., 2018; Smith et al., 2020; Nie et al., 2020; Xu et al., 2020a)), and…”