Classifying Political Orientation on Twitter: It's Not Easy!

By Raviv Cohen, Derek Ruths. 2013

275 citations recorded by the source corpus.

Numerous papers have reported great success at inferring the political orientation of Twitter users. This paper has some unfortunate news to deliver: while past work has been sound and often methodologically novel, we have discovered that reported accuracies have been systemically overoptimistic due to the way in which validation datasets have been collected, reporting accuracy levels nearly 30% higher than can be expected in populations of general Twitter users. Using careful and novel data collection and annotation techniques, we collected three different sets of Twitter users, each characterizing a different degree of political engagement on Twitter - from politicians (highly politically vocal) to "normal" users (those who rarely discuss politics). Applying standard techniques for inferring political orientation, we show that methods which previously reported greater than 90% inference accuracy, actually achieve barely 65% accuracy on normal users. We also show that classifiers cannot be used to classify users outside the narrow range of political orientation on which they were trained. While a sobering finding, our results quantify and call attention to overlooked problems in the latent attribute inference literature that, no doubt, extend beyond political orientation inference: the way in which datasets are assembled and the transferability of classifiers.

Subfields: Artificial Intelligence, Communication, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Sociology and Political Science

Citations in the corpus

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, methodology): “In the context of predicting users’ political orientations, researchers have shown that the choice of the proxy population drastically inﬂuences the performance of various prediction models (Cohen and Ruths, 2013).”
Beyond Binary Labels: Political Ideology Prediction of Twitter Users (2017, background): “However, these data sets are not representative samples of the entire population (Cohen and Ruths, 2013) and do not accurately reflect the variety of political attitudes and engagement (Kam et al., 2007).”
Studying User Income through Language, Behaviour and Affect in Social Media (2015, background): “In addition, recent research looked more deeply into classifying political orientation, analyzed data annotation and sampling biases in social media and showed how classification accuracy depends on the amount of data available for prediction [11, 31].”
Organizations Are Users Too: Characterizing and Detecting the Presence of Organizations on Twitter (2015, background): “Their dataset consists of tweets from 58 organizations, gathered from Twellow, and 600 personal accounts, identified by matching the account's profile description with a list of person names.”
Computational Sociolinguistics: A Survey (2016, background): “However, such an approach has the danger of creating biased datasets, which could influence the prediction performance (Cohen and Ruths 2013).”
Why Gender and Age Prediction from Tweets is Hard: Lessons from a Crowdsourcing Experiment (2014, background): “…gender identity (e.g. by sampling followers of accounts highly associated with males or females, such as sororities (Rao et al., 2010)), the results obtained on such a set may not be representative of a more random set (as observed when classifying political afﬁliation (Cohen and Ruths, 2013)).”
Discovering Biased News Articles Leveraging Multiple Human Annotations (2020, background): “…the scope of bias (Groseclose and Milyo, 2005), how it is generated (Gentzkow and Shapiro, 2010) and others that detect it in diﬀerent domains (Cohen and Ruths, 2013; Recasens et al., 2013; Iyyer et al., 2014), related work lacks automatic solutions for the binary clas-siﬁcation task that…”
#Snowden: Understanding Biases Introduced by Behavioral Differences of Opinion Groups on Social Media (2016, background): “Inferring Twitter users’ opinions is a known challenging task [17, 43].”
ConStance: Modeling Annotation Contexts to Improve Stance Classification (2017, background): “Still, researchers often decide what context to show annotators without rigorously evaluating how their decisions will affect annotations, on tasks from gender identiﬁcation to political leanings (Chen et al., 2015; Nguyen et al., 2014; Burger et al., 2011; Cohen and Ruths, 2013).”
Predictive Analysis on Twitter: Techniques and Applications (2018, unknown): “pinions of users based on linguistic characteristics (e.g., Tf-IDF) of their tweet content. While classication of users based on their political stance on Twitter has been well studied, Cohen et al. [91] have claimed that much of the studies and their datasets to date have covered very narrow portio”
Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey (2017, methodology): “[30] 2013 Evaluate the behavioral effects of different databases from political orientation classifiers -Political Figures Dataset -Politically Active Dataset -Politically Modest Dataset -Conover 2011 Dataset (C2D) [42] Introduce a topic model for contrastive opinion modeling Statement records of U.”