#TagSpace: Semantic Embeddings from Hashtags

By Jason Aaron Edward Weston, Sumit Chopra, Keith Adams. 2014

186 citations recorded by the source corpus.

We describe a convolutional neural network that learns feature representations for short textual posts using hashtags as a supervised signal. The proposed approach is trained on up to 5.5 billion words predicting 100,000 possible hashtags. As well as strong performance on the hashtag prediction task itself, we show that its learned representation of text (ignoring the hashtag labels) is useful for other tasks as well. To that end, we present results on a document recommendation task, where it also outperforms a number of baselines.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Ask the GRU: Multi-task Learning for Deep Text Recommendations (2016, neutral): “[43] proposed a CNN based model to predict hashtags on social media posts and found the learned representations to also be useful for document recommendation.”
PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks (2015, background): “However, when compared end-to-end with sophisticated deep learning approaches such as the convolutional neural networks (CNNs) [5, 8], the performance of text embeddings usually falls short on specific tasks [30].”
Representation learning for very short texts using weighted word embedding aggregation (2016, methodology): “…embeddings through a mean, max, min... function is still one of the most easy and widely used techniques to derive sentence embeddings, often in combination with an MLP or convolutional network (Weston et al. (2014); dos Santos and Gatti (2014); Yin and Sch¨utze (2015); Collobert et al. (2011)).”
Effective Use of Word Order for Text Categorization with Convolutional Neural Networks (2014, methodology): “…on token-level applications (e.g., POS tagging) by Collobert et al. (2011), CNN has been used in systems for entity search, sentence modeling, word embedding learning, product feature mining, and so on (Xu and Sarikaya, 2013; Gao et al., 2014; Shen et al., 2014; Kalchbrenner et a r X i v : 1412 .”
Attentive Pooling Networks (2016, methodology): “For example, a convolution (or a RNN) and a max-pooling is used to independently construct distributed vector representations of the input pair, followed by a large-margin training (Hu et al., 2014; Weston et al., 2014; Shen et al., 2014; dos Santos et al., 2015).”
Session-Based Social Recommendation via Dynamic Graph Attention Networks (2019, methodology): “on strategies that do not use an attention mechanism: a) averaging friends’ interests and; b) element-wise max-pooling over their interests—similar to techniques for aggregating word-level embeddings [36]. Mean aggregation outperforms the latter, but both are inferior to our proposed attention model”
LSTM-based Deep Learning Models for non-factoid answer selection (2015, unknown): “Following the same ranking loss in (Feng et al., 2015; Weston et al., 2014; Hu et al., 2014), we define the training objective as a hinge loss.”
Learning Semantic Similarity for Very Short Texts (2015, background): “Since then scientists have extensively used such embeddings to improve state-of-theart algorithms in natural language processing, such as partof-speech tagging [6], sentence completion [7], hashtag prediction [8], etc.”
Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding (2015, methodology): “On text, CNN has been gaining attention, used in systems for tagging, entity search, sentence modeling, and so on [4, 5, 26, 7, 21, 12, 25, 22, 24, 13], to make use of the 1D structure (word order) of text data.”
Classifying Relations by Ranking with Convolutional Neural Networks (2015, methodology): “, 2011), hashtag prediction (Weston et al., 2014), sentence completion and response matching (Hu et al.”
Improved Representation Learning for Question Answer Matching (2016, methodology): “It is important to note that the same biLSTM is applied to both q and a. Similar to (Feng et al., 2015; Weston et al., 2014; Hu et al., 2014), we define the training objective as a hinge loss.”
VisualBackProp: efficient visualization of CNNs (2018, background): “A plethora of important real-life problems are currently addressed with CNNs [1], including image recognition [2], speech recognition [3], and natural language processing [4].”
Applying deep learning to answer selection: A study and an open task (2015, methodology): “[6] use a CNN for hashtag prediction; Yu et al.”
User Conditional Hashtag Prediction for Images (2015, methodology): “The work of [18] is most similar to our work since we both propose hashtag embedding models trained with a ranking loss.”
VisualBackProp: visualizing CNNs for autonomous driving (2016, background): “A plethora of important real-life problems are currently addressed with CNNs (LeCun et al., 1989), including image recognition (Krizhevsky et al., 2012), speech recognition (Abdel-Hamid et al., 2012), and natural language processing (Weston et al., 2014).”
The Loss Surfaces of Multilayer Networks (2014, methodology): “Deep learning methods have enjoyed a resurgence of interest in the last few years for such applications as image recognition [Krizhevsky et al., 2012], speech recognition [Hinton et al., 2012], and natu-ral language processing [Weston et al., 2014].”