On Stopwords, Filtering and Data Sparsity for Sentiment Analysis of Twitter

By Hassan Saif, Miriam Fernández, Yulan He, Harith Alani. 2014

250 citations recorded by the source corpus.

Sentiment classification over Twitter is usually affected by the noisy nature (abbreviations, irregular forms) of tweets data. A popular procedure to reduce the noise of textual data is to remove stopwords by using pre-compiled stopword lists or more sophisticated methods for dynamic stopword identification. However, the effectiveness of removing stopwords in the context of Twitter sentiment classification has been debated in the last few years. In this paper we investigate whether removing stopwords helps or hampers the effectiveness of Twitter sentiment classification methods. To this end, we apply six different stopword identification methods to Twitter data from six different datasets and observe how removing stopwords affects two well-known supervised sentiment classification methods. We assess the impact of removing stopwords by observing fluctuations on the level of data sparsity, the size of the classifier’s feature space and its classification performance. Our results show that using pre-compiled lists of stopwords negatively impacts the performance of Twitter sentiment classification approaches. On the other hand, the dynamic generation of stopword lists, by removing those infrequent terms appearing only once in the corpus, appears to be the optimal method to maintaining a high classification performance while reducing the data sparsity and shrinking the feature space.

Subfields: Artificial Intelligence, Information Systems, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Sociology and Political Science

Citations in the corpus

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “, authorship and/or emotional states (Pennebaker et al., 2003; Saif et al., 2014), threatening as a result the research validity (Denny and Spirling, 2016).”
COVID-19 Public Sentiment Insights and MachineLearning for Tweets Classification (2020, methodology): “This customized replacement was in addition to the standard use of “Stopwords” and cleaning processes [48,49].”
Contextual semantics for sentiment analysis of Twitter (2016, methodology): “Note that we do not remove stopwords from tweets since they tend to carry sentiment information as shown in (Saif et al., 2014b,a).”
Like It or Not (2016, methodology): “Health Care Reform (HCR): The HCR dataset19 was presented by Speriosu et al. [2011] and consists of 2,515 manually annotated tweets of which 541 are positive, 1,381 negative, 470 neutral, 79 irrelevant, and 44 unsure tweets.”
A Brief Survey of Text Mining: Classification, Clustering and Extraction Techniques (2017, background): “tle information to distinguish different documents and also words occurring very rarely are also possibly of no significant relevance and can be removed from the documents [119, 130].”
Text Classification Algorithms: A Survey (2019, background): “The most common technique to deal with these words is to remove them from the texts and documents [42].”