Document Embedding with Paragraph Vectors

By Andrew M. Dai, Chris Olah, Christopher Olah, Quoc V. Le. 2015

382 citations recorded by the source corpus.

Paragraph Vectors has been recently proposed as an unsupervised method for learning distributed representations for pieces of texts. In their work, the authors showed that the method can learn an embedding of movie review texts which can be leveraged for sentiment analysis. That proof of concept, while encouraging, was rather narrow. Here we consider tasks other than sentiment analysis, provide a more thorough comparison of Paragraph Vectors to other document modelling algorithms such as Latent Dirichlet Allocation, and evaluate performance of the method as we vary the dimensionality of the learned representation. We benchmarked the models on two document similarity data sets, one from Wikipedia, one from arXiv. We observe that the Paragraph Vector method performs significantly better than other methods, and propose a simple improvement to enhance embedding quality. Somewhat surprisingly, we also show that much like word embeddings, vector operations on Paragraph Vectors can perform useful semantic results.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

TensorFlow: A system for large-scale machine learning (2016, neutral): “, a large language model may use over 10(9) parameters with a vocabulary of 800,000 words [39], and we have experience with document models [20] where the parameters occupy several terabytes.”
Evaluation of Text Generation: A Survey (2020, background): “By only measuring word distances, the metric cannot capture information conveyed in the group of words, for which we need higher-level document representations (Dai et al., 2015; Wu et al., 2018).”
Latent Cross: Making Use of Context in Recurrent Recommender Systems (2018, background): “• Paragraph Vector (PV): Use [13] to learn unsupervised embeddings of each user (based on user demographics and previous watches).”
Neural information retrieval: at the end of the early years (2017, methodology): “…search Aggregate Explicit Kim et al. (2016), Kusner et al. (2014) 9.1.2 Learn Learn to autoencode Salakhutdinov and Hinton (2009) Learn to predict Dai et al. (2014), Djuric et al. (2015), Le and Mikolov (2014) 9.2 Detecting text reuse Aggregate Explicit Zhang et al. (2014) 9.3.1 Similar Question…”
Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts (2019, background): “By only measuring word distances, the metric cannot capture information conveyed by the grouping of words, for which we need higher-level document representations (Dai et al., 2015; Wu et al., 2018).”
Contextual LSTM (CLSTM) models for Large scale NLP tasks (2016, background): “Paragraph vectors [8, 26] propose an unsupervised algorithm that learns a latent variable from a sample of words from the context of a word, and uses the learned latent context representation as an auxiliary input to an underlying skip-gram or Continuous Bag-of-words (CBOW) model.”
Sentiment Classification Using Document Embeddings Trained with Cosine Similarity (2019, methodology): “Secondly, as cosine similarity is widely used to measure document similarity (Singhal, 2001; Dai et al., 2015), we believe our method should more directly maximize the cosine similarity be-tween similar document vectors.”