Language as a Latent Variable: Discrete Generative Models for Sentence Compression

By Yishu Miao, Phil Blunsom. 2016

228 citations recorded by the source corpus.

In this work we explore deep generative models of text in which the latent representation of a document is itself drawn from a discrete language model distribution. We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences. In this application the generative model first draws a latent summary sentence from a background language model, and then subsequently draws the observed sentence conditioned on this latent summary. In our empirical evaluation we show that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data. Further, we explore semi-supervised compression scenarios where we show that it is possible to achieve performance competitive with previously proposed supervised models while training on a fraction of the supervised data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “, 2016), variational autoencoders (Miao and Blunsom, 2016), and direct optimization of the performance metric (Ranzato et al.”
Learning with Latent Language (2017, background): “String-valued latent variables are widely used in language processing tasks ranging from morphological analysis (Dreyer and Eisner, 2009) to sentence compression (Miao and Blunsom, 2016).”
Revisiting Self-Training for Neural Sequence Generation (2019, methodology): “…et al., 2015) makes use of the monolingual data on the target side to improve machine translation systems, latent variable models (Kingma et al., 2014) are employed to incorporate unlabeled source data to facilitate sentence compression (Miao & Blunsom, 2016) or code generation (Yin et al., 2018).”
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents (2018, background): “, 2016) and later extended by source copying (Miao and Blunsom, 2016; See et al., 2017), reinformcement learning”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, background): “• Language compression is the goal of ASC ( Autoen-coding Sentence Compression ), a model introduced in [124].”
Abstractive Summarization of Reddit Posts with Multi-level Memory Networks (2018, background): “Some notable examples include the use of variational autoencoders (VAEs) (Miao and Blunsom, 2016; Li et al., 2017), graph-based attention (Tan et al., 2017), pointer-generator models (See et al., 2017), self-attention networks (Liu et al., 2018), reinforcement learning (Paulus et al., 2018;…”
Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting (2018, background): “…hierarchical attention (Nallapati et al., 2016), coverage (Suzuki and Nagata, 2016; Chen et al., 2016; See et al., 2017), RL based metric optimization (Paulus et al., 2018), graph-based attention (Tan et al., 2017), and the copy mechanism (Miao and Blunsom, 2016; Gu et al., 2016; See et al., 2017).”
Latent Intention Dialogue Models (2017, methodology): “This is critical for NLP tasks, especially where additional supervision and external knowledge can be utilized for bootstrapping (Faruqui et al., 2015; Miao & Blunsom, 2016; Koˇcisk´y et al., 2016).”
Extractive Summarization as Text Matching (2020, background): “and Favre, 2009), reconstruction (Miao and Blunsom, 2016) and maximize semantic volume (Yogatama et al.”
StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing (2018, methodology): “Such a structural latent space is contrast to existing VAE research using flat representations, such as continuous distributed representations (Kingma and Welling, 2013), discrete symbols (Miao and Blunsom, 2016), or hybrids of the two (Zhou and Neubig, 2017).”
Intra-agent speech permits zero-shot task acquisition (2022, background): “VAEs with language latent variables have also been studied in language summarization [47], and in the context of multi-agent communication with natural language [48, 49].”
Multi-document Summarization via Deep Learning Techniques: A Survey (2020, methodology): “This network is inspired by pointer network [128], copynet [44], forced-attention sentence compression [81] and coverage mechanism from machine translation [123].”
SEQˆ3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression (2019, methodology): “components, unlike works that rely on reinforcement learning (Miao and Blunsom, 2016; Wang and Lee, 2018).”
MeanSum: A Neural Model for Unsupervised Multi-Document Abstractive Summarization (2018, background): “Miao & Blunsom (2016) train an auto-encoder to do extractive sentence compression and combines it with a model trained on parallel data to do semi-supervised summarization.”
Joint Parsing and Generation for Abstractive Summarization (2019, methodology): “(2015); • RAS-LSTM and RAS-Elman (Chopra, Auli, and Rush 2016) describe a convolutional attentive encoder that ensures the decoder focuses on appropriate words at each step of generation; • ASC+FSC1 (Miao and Blunsom 2016) presents a generative auto-encoding sentence compression model jointly traine”