An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation

By Jey Han Lau, Timothy Baldwin. 2016

708 citations recorded by the source corpus.

Recently, Le and Mikolov (2014) proposed doc2vec as an extension to word2vec (Mikolov et al., 2013a) to learn document-level embeddings. Despite promising results in the original paper, others have struggled to reproduce those results. This paper presents a rigorous empirical evaluation of doc2vec over two tasks. We compare doc2vec to two baselines and two state-of-the-art document embedding methodologies. We found that doc2vec performs robustly when using models trained on large external corpora, and can be further improved by using pre-trained word embeddings. We also provide recommendations on hyper-parameter settings for general purpose applications, and release source code to induce document embeddings using our trained doc2vec models.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, methodology): “…semantic representations (i.a., Arora et al. (2017); Conneau et al. (2017); Mu et al. (2017); Pagliardini et al. (2017); Wieting and Gimpel (2017); He and Lin (2016); Hill et al. (2016); Kenter et al. (2016); Lau and Baldwin (2016); Wieting et al. (2016a,b); He et al. (2015); Pham et al. (2015)).”
It’s All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution (2019, methodology): “Le and Mikolov (2014), we use a standard sentiment clas-siﬁcation task to quantify the downstream performance of the embedding spaces when they are used as a pretrained word embedding input (Lau and Baldwin, 2016) to Doc2Vec on the Stanford Large Movie Review dataset.”
Learning Semantic Textual Similarity from Conversations (2018, methodology): “(2) For model comparisons, we include the state-of-the-art neural STS model CNN (HCTI) (Shao, InferSent (Conneau et al., 2017), Sent2Vec (Pagliardini et al., 2017), SIF (Arora et al., 2017), PV-DBOW (Lau and Baldwin, 2016), C-PHRASE (Kruszewski et al., 2015), ECNU (Tian et al., 2017) and BIT (Wu et ”
Community-level Research on Suicidality Prediction in a Secure Environment: Overview of the CLPsych 2021 Shared Task (2021, positive): “Latent features included using Doc2vec embeddings (Lau and Baldwin, 2016).”
Scalable Deep Multimodal Learning for Cross-Modal Retrieval (2019, methodology): “The 300-dimensional text representation is extracted by a Doc2Vec model5 [16], which is pre-trained on Wikipedia.”
SPECTER: Document-level Representation Learning using Citation-informed Transformers (2020, methodology): “We trained Doc2Vec on our training subset using Gensim (Řehůřek and Sojka, 2010), and chose the hyperparameter grid using suggestions from Lau and Baldwin (2016).”
Multichannel Attention Refinement for Video Question Answering (2020, methodology): “…are encoded by just averaging the GloVe word embeddings; (2) D2V: the sentences are encoded using the Doc2Vec model pre-trained on Wikipedia [26]; (3) BERT: the sentences are encoded using the pre-trained BERT model, and outputs of the last two layers are summed to get the embedding of the…”
Predictive Analysis on Twitter: Techniques and Applications (2018, methodology): “The multi-entry neural network architecture (MENET) developed for location prediction uses words, the semantics of the paragraph (using doc2vec [79]), network features and topology (using node2vec [80]) and time-stamps to deduce users location.”
Sentiment Classification Using Document Embeddings Trained with Cosine Similarity (2019, neutral): “We did not tune the N hyperparameter or the negative sampling size and left it the same as in (Li et al., 2016a) and (Lau and Baldwin, 2016).”
Top2Vec: Distributed Representations of Topics (2020, background): “The doc2vec model outperforms BOW models and produces state-of-the-art results on many linguistics tasks compared to traditional methods [17, 18].”
Multi-View Domain Adapted Sentence Embeddings for Low-Resource Unsupervised Duplicate Question Detection (2019, background): “Lau and Baldwin (2016) evaluate two representation learning techniques (doc2vec (Le and Mikolov, 2014) and word2vec (Mikolov et al., 2013a)) on CQADupStack.”