Distributed Representations of Sentences and Documents
By Quoc V. Le, Tomas Mikolov. 2014
9,778 citations recorded by the source corpus.
Many machine learning algorithms require the input to be represented as a fixed-length feature vector. When it comes to texts, one of the most common fixed-length features is bag-of-words. Despite their popularity, bag-of-words features have two major weaknesses: they lose the ordering of the words and they also ignore semantics of the words. For example, "powerful," "strong" and "Paris" are equally distant. In this paper, we propose Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents. Our algorithm represents each document by a dense vector which is trained to predict words in the document. Its construction gives our algorithm the potential to overcome the weaknesses of bag-of-words models. Empirical results show that Paragraph Vectors outperforms bag-of-words models as well as other techniques for text representations. Finally, we achieve new state-of-the-art results on several text classification and sentiment analysis tasks.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Yoshua Bengio
- Graham Neubig
- Kyunghyun Cho
- Quoc V. Le
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Yejin Choi
- Percy Liang
- Noah A. Smith
In Representation & Self-Supervised Learning
- Yoshua Bengio
- Sebastian Ruder
- Ruslan Salakhutdinov
- Quoc V. Le
- Jianfeng Gao
- James R. Glass
- Honglak Lee
- Kyunghyun Cho
Citations in the corpus
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “, 2015; Logeswaran and Lee, 2018) or paragraph embeddings (Le and Mikolov, 2014).”
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, neutral): “Beyond multi-task learning, much work towards developing general NLU systems has focused on sentence-to-vector encoder functions (Le and Mikolov, 2014; Kiros et al., 2015, i.a.), leveraging unlabeled data (Hill et al., 2016; Peters et al., 2018), labeled data (Conneau and Kiela, 2018; McCann et…”
- An efficient framework for learning sentence representations (2018, methodology): “There have been variations on the decoder such as autoencoder models which predict the input sentence instead of neighboring sentences (Hill et al., 2016) and predicting properties of a window of words in the input sentence (Le & Mikolov, 2014).”
- SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, background): “0 PV-DBOW Paragraph vectors (PV-DBOW) (Le and Mikolov, 2014; Lau and Baldwin, 2016) 72.”
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “Recent work on generating sentence embeddings range from models that compose word embeddings (Le and Mikolov, 2014; Arora et al., 2017; Wieting et al., 2016) to more complex neural network architectures.”
- Semi-supervised Sequence Learning (2015, background): “The previous baselines are bag-of-words, ConvNets [13] or Paragraph Vectors [19].”
- Skip-Thought Vectors (2015, methodology): “The paragraph vector of [7] is an alternative to the above models in that it can learn unsupervised sentence representations by introducing a distributed sentence indicator as part of a neural language model.”
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “Learning representations of natural language has been shown to be useful for a wide range of NLP tasks and has been widely adopted (Mikolov et al., 2013; Le & Mikolov, 2014; Dai & Le, 2015; Peters et al., 2018; Devlin et al., 2019; Radford et al., 2018; 2019).”
- MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “…et al., 2006; Collobert & Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) and sentence-level representations (Kiros et al., 2015; Logeswaran & Lee, 2018; Le & Mikolov, 2014), as well as context sensitive features from the NMT model (McCann et al., 2017) and ELMo (Peters et al., 2018).”
- Adaptive Computation Time for Recurrent Neural Networks (2016, background): “Firstly, it has been observed that the high-dimensional representations present in neural networks naturally tend to behave in a linear way [32, 20], even remaining consistent under arithmetic operations such as addition and subtraction [22].”
- Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, background): “, 2014b), parse tree structure (Irsoy and Cardie, 2014), and run-time inference (Le and Mikolov, 2014).”
- A Neural Network for Factoid Question Answering over Paragraphs (2014, methodology): “More relevant to this work are the dialogue analysis model proposed by Kalchbren-ner & Blunsom (2013) and the paragraph vector model described in Le and Mikolov (2014), both of which are able to generate distributed representations of paragraphs.”
- Parameter-Efficient Transfer Learning for NLP (2019, background): “These embeddings may be at the word (Mikolov et al., 2013), sentence (Cer et al., 2019), or paragraph level (Le & Mikolov, 2014).”
- XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “Sentence Representation Learning Many approaches have been proposed to extend word embeddings to sentence or paragraph representations (Le and Mikolov, 2014; Wieting et al., 2016; Arora et al., 2017).”