PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks

By Jian Tang, Meng Qu, Qiaozhu Mei. 2015

818 citations recorded by the source corpus.

Unsupervised text embedding methods, such as Skip-gram and Paragraph Vector, have been attracting increasing attention due to their simplicity, scalability, and effectiveness. However, comparing to sophisticated deep learning architectures such as convolutional neural networks, these methods usually yield inferior results when applied to particular machine learning tasks. One possible reason is that these text embedding methods learn the representation of text in a fully unsupervised way, without leveraging the labeled information available for the task. Although the low dimensional representations learned are applicable to many different tasks, they are not particularly tuned for any task. In this paper, we fill this gap by proposing a semi-supervised representation learning method for text data, which we call the predictive text embedding (PTE). Predictive text embedding utilizes both labeled and unlabeled data to learn the embedding of text. The labeled information and different levels of word co-occurrence information are first represented as a large-scale heterogeneous text network, which is then embedded into a low dimensional space through a principled and efficient algorithm. This low dimensional embedding not only preserves the semantic closeness of words and documents, but also has a strong predictive power for the particular task. Compared to recent supervised approaches based on convolutional neural networks, predictive text embedding is comparable or more effective, much more efficient, and has fewer parameters to tune.

Subfields: Artificial Intelligence, Materials Chemistry, Natural Language Processing, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Natural Language Processing

In Graph Learning

Citations in the corpus

Graph Convolutional Networks for Text Classification (2018, background): “• PTE : predictive text embedding (Tang, Qu, and Mei 2015), which ﬁrstly learns word embedding based on heterogeneous text network containing words, documents and labels as nodes, then averages word embeddings as document embeddings for text classiﬁcation.”
Joint Embedding of Words and Labels for Text Classification (2018, methodology): “In NLP, labels embedding for text classification has been studied in the context of heterogeneous networks in (Tang et al., 2015) and multitask learning in (Zhang et al.”
A Comprehensive Survey of Graph Embedding: Problems, Techniques, and Applications (2017, neutral): “The example classifiers include SVM ([1], [20], [33], [34], [41], [42], [45], [56], [57], [60], [73], [75], [81], [87]), logistic regression ([1], [17], [19], [20], [21], [25], [27], [28], [45], [59], [124]) and k-nearest neighbour classification”
Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2vec (2017, methodology): “In PTE, { α , β , γ } satisfy α vol ( G ww ) = β vol ( G dw ) = γ vol ( G lw ) . is because the authors perform edge sampling in training where edges from three different networks are sampled alternatively (see Section 4.2 in [34]).”
Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking (2017, methodology): “LINE (Tang et al., 2015b) uses ﬁrst-and second-order proximity and trains the embedding via negative sampling.”
Heterogeneous Information Network Embedding for Recommendation (2017, background): “Recently, several works [54], [55], [56], [57], [58] attempt to analyze heterogeneous networks via embedding methods.”
GraphGAN: Graph Representation Learning with Generative Adversarial Nets (2017, methodology): “roposed GraphGAN with the following four baselines for graph representation learning: DeepWalk (Perozzi, Al-Rfou, and Skiena 2014) adopts random walk and Skip-Gram to learn vertex embeddings. LINE (Tang et al. 2015) preserves the ﬁrst-order and second-order proximity among vertices in the graph. Nod”
Representation Learning for Attributed Multiplex Heterogeneous Network (2019, background): “Heterogeneous Network (HEN) PTE [34] Multi Single / metapath2vec [7] HERec [31] Attributed HEN (AHEN) HNE [3] Multi Single Attributed”
CGLB: Benchmark Tasks for Continual Graph Learning (2022, methodology): “…Network (HAN) [49], Heterogeneous Graph Transformer (HGT) [22], Relational Graph Convolutional Networks (R-GCNs) [41], Metapath2vec [11], and Predictive Text Embedding (PTE) [44] will be implemented into our continual learning pipelines, which will be benchmarked with the constructed tasks.”
SNE: Signed Network Embedding (2017, background): “The heterogeneous network embedding models are studied in [4, 25, 29, 37].”
A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources (2020, neutral): “PTE [56] divides the documents into word-word graph, word-document graph and word-label graph.”
metapath2vec: Scalable Representation Learning for Heterogeneous Networks (2017, methodology): “Method PathSim [26] DeepWalk / node2vec [8, 22] LINE (1st+2nd) [30] PTE [29] metapath2vec metapath2vec++”
Fast Network Embedding Enhancement via High Order Proximity Approximation (2017, methodology): “DeepWalk [Perozzi et al., 2014] employs Skip-gram [Mikolov et al., 2013] model, which is originally used in word representation learning, on random walks for NRL. LINE [Tang et al., 2015b] models first-order and second-order proximities between vertices for learning large-scale network embeddings.”