Learning Generic Sentence Representations Using Convolutional Neural Networks

By Zhe Gan, Yunchen Pu, Ricardo Henao, Chunyuan Li, Xiaodong He, Lawrence Carin. 2016

101 citations recorded by the source corpus.

We propose a new encoder-decoder approach to learn distributed sentence representations that are applicable to multiple purposes. The model is learned by using a convolutional neural network as an encoder to map an input sentence into a continuous vector, and using a long short-term memory recurrent neural network as a decoder. Several tasks are considered, including sentence reconstruction and future sentence prediction. Further, a hierarchical encoder-decoder model is proposed to encode a sentence to predict multiple future sentences. By training our models on a large collection of novels, we obtain a highly generic convolutional sentence encoder that performs well in practice. Experimental results on several benchmark datasets, and across a broad range of applications, demonstrate the superiority of the proposed model over competing methods.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

An efficient framework for learning sentence representations (2018, methodology): “Drawing inspiration from this model, Gan et al. (2016) explore the use of convolutional neural network (CNN) encoders.”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “Other objectives for sentence embedding learning include predicting future sentences rather than only neighbors (Gan et al., 2017) and predicting explicit discourse markers (Jernite et al., 2017; Nie et al., 2019).”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, methodology): “For this backward model q φ p S | T q , we use the same CNN-LSTM architecture as the forward model [20].”
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms (2018, background): “The Convolutional Neural Network (CNN) architecture (Kim, 2014; Collobert et al., 2011; Gan et al., 2017; Zhang et al., 2017b; Shen et al., 2018) is another strategy extensively employed as the compositional function to encode text sequences.”
Adversarial Feature Matching for Text Generation (2017, methodology): “Toward this end, we initialize the LSTM parameters of the generator by pre-training a standard CNN-LSTM autoencoder (Gan et al., 2016).”
ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, background): “, 2016), encoder-decoder architectures (Kiros et al., 2015; Gan et al., 2017), and other sources of supervision and learning frameworks (Le and Mikolov, 2014; Pham et al.”
Deconvolutional Paragraph Representation Learning (2017, methodology): “The proposed approach uses a Convolutional Neural Network (CNN) [21, 22, 23] as encoder and a deconvolutional (i.”
On the Use of Word Embeddings Alone to Represent Natural Language Sequences (2018, unknown): “The Convolutional Neural Network (CNN) architecture in Kim 2014; Collobert et al. 2011; Gan et al. 2017 is another strategy extensively employed as the compositional function for encoding text sequences.”
An Unsupervised Sentence Embedding Method by Mutual Information Maximization (2020, background): “unlabeled or partially labeled dataset to address the aforementioned challenge (Kiros et al., 2015; Gan et al., 2017; Hill et al., 2016; Pagliardini et al., 2017; Yang et al., 2018).”
JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets (2018, methodology): “The caption features are obtained from a pretrained word-level CNN-LSTM autoencoder (Gan et al., 2017b), which aims to achieve a one-to-one mapping between the captions and the features.”
Learning Context-Sensitive Convolutional Filters for Text Processing (2017, background): “, 2017), generic text representations (Gan et al., 2016; Tang et al., 2018), language modeling (Dauphin et al.”
Towards Better Representations with Deep/Bayesian Learning (2018, methodology): “In the traditional factor analysis (Gan et al., 2017c), the linear relation between x and z is assumed, and inference is done using mean field VB or Gibbs sampling.”
SLM: Learning a Discourse Language Representation with Sentence Unshuffling (2020, methodology): “Inspired by prior works about sentence-based representations for recurrent networks (Kiros et al., 2015; Hill et al., 2016; Gan et al., 2017; Jernite et al., 2017; Logeswaran et al., 2018; Gong et al., 2016; Chen et al., 2016), we seek to incorporate a more explicit hierarchy into the transformer by”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “Another example is sentence embeddings, which are often evaluated on sentence classification and sentence similarity tasks [80, 116, 147, 180, 201].”
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors (2018, methodology): “We then apply the à la carte method to two tasks measuring the ability to learn new or synthetic words from context, achieving strong results on the nonce task of Herbelot and Baroni (2017).”