Deconvolutional Paragraph Representation Learning

By Yizhe Zhang, Dinghan Shen, Guoyin Wang, Zhe Gan, Ricardo Henao, Lawrence Carin. 2017

79 citations recorded by the source corpus.

Learning latent representations from long text sequences is an important first step in many natural language processing applications. Recurrent Neural Networks (RNNs) have become a cornerstone for this challenging task. However, the quality of sentences during RNN-based decoding (reconstruction) decreases with the length of the text. We propose a sequence-to-sequence, purely convolutional and deconvolutional autoencoding framework that is free of the above issue, while also being computationally efficient. The proposed method is simple, easy to implement and can be leveraged as a building block for many applications. We show empirically that compared to RNNs, our framework is better at reconstructing and correcting long paragraphs. Quantitative evaluation on semi-supervised text classification and summarization tasks demonstrate the potential for better utilization of long unlabeled text data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, methodology): “used in TF-IDF or BM25, dense paragraph vectors learned by deep neural networks (Zhang et al., 2017; Conneau et al., 2017) can capture much richer semantics beyond the n-gram term features.”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, background): “The ﬁlter size, stride and the word embedding dimension were set to 5 , 2 and 300 , respectively, following [46].”
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms (2018, background): “…et al., 2015), to more sophisticated compositional functions such as Recurrent Neural Networks (RNNs) (Tai et al., 2015; Sutskever et al., 2014), Convolutional Neural Networks (CNNs) (Kalchbrenner et al., 2014; Kim, 2014; Zhang et al., 2017a) and Recursive Neural Networks (Socher et al., 2011a).”
Joint Embedding of Words and Labels for Text Classification (2018, methodology): “Recently, neural models have been employed to learn text representations, including convolutional neural networks (CNNs) (Kalchbrenner et al., 2014; Zhang et al., 2017b; Shen et al., 2017) and recurrent neural networks (RNNs) based on long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997;…”
How to Fine-Tune BERT for Text Classification? (2019, neutral): “…text representation, including convolution models (Kalchbrenner et al., 2014; Zhang et al., 2015; Conneau et al., 2016; Johnson and Zhang, 2017; Zhang et al., 2017; Shen et al., 2018), recurrent models (Liu et al., 2016; Yogatama et al., 2017; Seo et al., 2017), and attention mechanisms (Yang…”
Adversarial Symmetric Variational Autoencoder (2017, background): “Variational autoencoders (VAEs) are designed to learn both an encoder and decoder, leading to excellent data reconstruction and the ability to quantify a bound on the log-likelihood fit of the model to data [12, 13, 14, 15, 16, 17, 18, 19].”
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing (2019, methodology): “To clearly compare the quality of z , we build a simple one-layer classiﬁer on z , and ﬁne-tune the model on different proportions of labelled data (Zhang et al., 2017b).”
Towards Better Representations with Deep/Bayesian Learning (2018, methodology): “A popular method is to use Stochastic Gradient Markov Chain Monte Carlo (SG-MCMC) methods to generate posterior samples (Welling and Teh, 2011; Chen et al., 2014; Ding et al., 2014; Li et al., 2016a; Zhang et al., 2017c).”
Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignment (2018, background): “In future work, we aim to leverage abundant unlabeled text data to abstract more informative sentence representations (Dai and Le, 2015; Zhang et al., 2017; Shen et al., 2017; Tang and de Sa, 2018) .”
NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing (2018, unknown): “, 2014) or DCNN-based autoencoder (Zhang et al., 2017).”
Dual Latent Variable Model for Low-Resource Natural Language Generation in Dialogue Systems (2018, methodology): “To solve the exposure-bias problem (Bengio et al., 2015) Zhang et al. (2017); Shen et al. (2017) proposed a seq2seq purely convolutional and deconvolutional autoencoder, Yang et al. (2017) proposed to use a dilated CNN decoder in a latentvariable model, or Semeniuta et al. (2017) proposed a hybrid…”