Learning Neural Templates for Text Generation

By Sam Wiseman, Stuart M. Shieber, Alexander M. Rush. 2018

207 citations recorded by the source corpus.

While neural, encoder-decoder models have had significant empirical success in text generation, there remain several unaddressed problems with this style of generation. Encoder-decoder models are largely (a) uninterpretable, and (b) difficult to control in terms of their phrasing or content. This work proposes a neural generation system using a hidden semi-markov model (HSMM) decoder, which learns latent, discrete templates jointly with learning to generate. We show that this model learns useful templates, and that these templates make generation both more interpretable and controllable. Furthermore, we show that this approach scales to real data sets and achieves strong performance nearing that of encoder-decoder text generation models.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Some other notable directions in this space involve fact-aware pretraining (Falke et al., 2019; Wan and Bansal, 2022) and structured learning frameworks (Wiseman et al., 2018).”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, background): “Diversity-promoting objectives make it possible to generate diverse natural language descriptions (Li et al., 2016; Wiseman et al., 2018).”
Skill Induction and Planning with Latent Language (2021, methodology): “The structure of this dynamic program is identical to the forward algorithm for hidden semi-Markov models (HSMMs), which are widely used in NLP for tasks like language generation and word alignment (Wiseman et al., 2018).”
Few-Shot NLG with Pre-Trained Language Model (2019, methodology): “…based approaches have been used, including the end-to-end methods that jointly model context selection and surface realization (Liu et al. 2018; Wiseman, Shieber, and Rush 2018; Puduppully, Dong, and Lapata 2018), and the methods using neural network to tackle realization component (Moryossef,…”
Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialogue (2019, methodology): “We then used the Viterbi segmentations from the model released by Wiseman et al. (2018) to get spans corresponding to each argument.”
CTRLsum: Towards Generic Controllable Text Summarization (2020, neutral): “, 2019), and templates (Guu et al., 2018; Wiseman et al., 2018; He et al., 2020a).”
Controlling hallucinations at word level in data-to-text generation (2021, unknown): “– hsmm [61], an encoder-decoder model with a multi-branch decoder.”
Retrieval-guided Dialogue Response Generation via a Matching-to-Generation Framework (2019, methodology): “Wiseman et al. (2017, 2018) used either ﬁxed template or learned templates for data-to-text generation.”
Can Neural Generators for Dialogue Learn Sentence Planning and Discourse Structuring? (2018, methodology): “the WebNLG challenge or WikiBio dataset (Wiseman et al., 2018; Colin et al., 2016).”
Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training (2021, background): “, 2019), latent templates for interpretability (Wiseman et al., 2018) and text-to-text generation with T5 (Kale, 2020).”
AggGen: Ordering and Aggregating while Generating (2021, methodology): “AGGGEN outperforms NTemp+AR and is comparable with Shen et al. (2020), but performs slightly worse than both seq2seq models in terms of wordoverlap metrics.”
Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation (2019, background): “…et al., 2019), reinforcement learning-based rewards (Paulus et al., 2018; Pasunuru & Bansal, 2018), semi-Markov models to learn neural templates (Wiseman et al., 2018), data augmentation (Ma et al., 2019; Kedzie & McKeown, 2019), content planning (Puduppully et al., 2019a), and constrained…”
Large Scale Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training (2020, methodology): “…been developed for these tasks, and evaluated on these benchmark datasets, such as CRFs (Lebret et al., 2016), latent templates for interpretability (Wiseman et al., 2018), graph trans-formers over structured data (Koncel-Kedziorski et al., 2019), and text-to-text generation with T5 (Kale, 2020).”
Latent Relation Language Models (2019, background): “Latent structure has included predicting latent sequences of topics (Wiseman et al., 2018), chunking of word sequences into n -grams (Buckman and Neu-big, 2018), deciding between input sources (Ling et al., 2016; Gu et al., 2016), predicting latent continuous vectors (Bowman et al., 2016),…”
Low-Resource Generation of Multi-hop Reasoning Questions (2020, background): “The low-resource problem has been studied in the tasks such as machine translation (Gu et al., 2018), pos tagging (Kann et al., 2018), word embedding (Jiang et al., 2018), text generation (Wiseman et al., 2018), and dialogue systems (Mi et al., 2019).”
Encode, Tag, Realize: High-Precision Text Editing (2019, background): “Recent work discusses some of the difficulties of learning neural decoders for text generation (Wiseman et al., 2018; Prabhakaran et al., 2018).”
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020, methodology): “Explicit syntactically-controlled text generation has been studied previously using labels such as constituency parse templates (Iyyer et al., 2018; Akoury et al., 2019) or learned discrete latent templates (Wiseman et al., 2018).”