Interpretation of Semantic Tweet Representations

By Ganesh Jawahar, Manish Gupta, Vasudeva Varma. 2017

8 citations recorded by the source corpus.

Research in analysis of microblogging platforms is experiencing a renewed surge with a large number of works applying representation learning models for applications like sentiment analysis, semantic textual similarity computation, hashtag prediction, etc. Although the performance of the representation learning models has been better than the traditional baselines for such tasks, little is known about the elementary properties of a tweet encoded within these representations, or why particular representations work better for certain tasks. Our work presented here constitutes the first step in opening the black-box of vector embeddings for tweets.Traditional feature engineering methods for high-level applications have exploited various elementary properties of tweets. We believe that a tweet representation is effective for an application because it meticulously encodes the application-specific elementary properties of tweets. To understand the elementary properties encoded in a tweet representation, we evaluate the representations on the accuracy to which they can model each of those properties such as tweet length, presence of particular words, hashtags, mentions, capitalization, etc.Our systematic extensive study of nine supervised and four unsupervised tweet representations against most popular eight textual and five social elementary properties reveal that Bi-directional LSTMs (BLSTMs) and Skip-Thought Vectors (STV) best encode the textual and social properties of tweets respectively. FastText is the best model for low resource settings, providing very little degradation with reduction in embedding size. Finally, we draw interesting insights by correlating the model performance obtained for elementary property prediction tasks with the high- level downstream applications.

Subfields: Artificial Intelligence, Natural Language Processing, Sentiment & Opinion Analysis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Sentiment & Opinion Analysis

Yonatan Belinkov

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “(Ganesh et al., 2017) Sentence embeddings Sentence length, word presence, word order; orthography; social tasks Classification”
Identifying and Controlling Important Neurons in Neural Machine Translation (2018, background): “…neural representations of linguistic units, such as word embeddings (K ¨ ohn, 2015; Qian et al., 2016b), sentence embeddings (Adi et al., 2016; Ganesh et al., 2017; Brunner et al., 2018), and NMT representations at different linguistic levels: morphological (Belinkov et al., 2017), syntactic…”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “eludes word embeddings [138, 188, 279], RNN hidden states or gate activations [102, 278, 306, 335, 345], and sentence embeddings [2, 3, 47, 81, 106, 117].”
One Size Does Not Fit All: Comparing NMT Representations of Different Granularities (2019, background): “This includes analyzing word and sentence embeddings (Adi et al., 2017; Qian et al., 2016b; Ganesh et al., 2017; Conneau et al., 2018, among others), RNN states (Qian et al., 2016a; Shi et al., 2016; Wu and King, 2016; Wang et al., 2017), and NMT representations (Shi et al., 2016; Belinkov et al.,…”