Big Questions for Social Media Big Data: Representativeness, Validity and Other Methodological Pitfalls

By Zeynep Tufekci. 2014

765 citations recorded by the source corpus.

Large-scale databases of human activity in social media have captured scientific and policy attention, producing a flood of research and discussion. This paper considers methodological and conceptual challenges for this emergent field, with special attention to the validity and representativeness of social media big data analyses. Persistent issues include the over-emphasis of a single platform, Twitter, sampling biases arising from selection by hashtags, and vague and unrepresentative sampling frames. The socio-cultural complexity of user behavior aimed at algorithmic invisibility (such as subtweeting, mock-retweeting, use of “screen captures” for text, etc.) further complicate interpretation of big data social media. Other challenges include accounting for field effects, i.e. broadly consequential events that do not diffuse only through the network under study but affect the whole society. The application of network methods from other fields to the study of human social activity may not always be appropriate. The paper concludes with a call to action on practical steps to improve our analytic capacity in this promising, rapidly-growing field.

Subfields: Sociology and Political Science, Statistical and Nonlinear Physics, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Sociology and Political Science

In Statistical and Nonlinear Physics

In AI Ethics, Fairness & Society

Citations in the corpus

A Survey on Bias and Fairness in Machine Learning (2019, background): “An example can be observed in Twitter where people talking about a particular topic start using a hashtag at some point to capture attention, then continue the discussion about the event without using the hashtag [99, 120].”
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, methodology): “Results of various methods to collect, measure, or process data should be routinely juxtaposed (Ruths and Pfeﬀer, 2014; Tufekci, 2014).”
Bringing the People Back In: Contesting Benchmark Machine Learning Datasets (2020, background): “For instance, in her critique of Twitter as one of the most common model organisms, the fruit fly (or drosophila melanogaster) of large-scale social media research, Tufekci (2014) points to how such a focus obscures more complicated social processes at work, as the particular technological affordanc”
Trustworthy Artificial Intelligence: A Review (2022, background): “Data bias also occurs if the population demographics of the training dataset are entirely different from the target population on which the system is being implemented [195].”
Collective aspects of privacy in the Twitter social network (2018, background): “Further research should try with other current social networks to avoid the Twitter model organism bias [34] and to ensure that our knowledge applies to the online society and not only a handful of social networking sites.”
Social Media data: Challenges, opportunities and limitations in urban studies (2019, neutral): “…sample of citizen preferences, opinions and activities (Agryzkov et al., 2015; Barbera & Rivero, 2015; Martí, Serrano-Estrada, & Nolasco-Cirugeda, 2017; Morstatter, Pfeffer, Liu, & Carley, 2013; Tufekci, 2014), given the increasing diversity of user profiles (Pew Research Center, 2017).”
Identifying Platform Effects in Social Media Data (2016, background): “This concern has thus far only been raised theoretically (Tufekci 2014; Ruths and Pfeffer 2014), and not yet addressed empirically.”
Data ex Machina: Introduction to Big Data (2017, background): “The first is to view these platforms as generalizable microcosms of society (Tufekci 2014).”
When Are Search Completion Suggestions Problematic? (2020, background): “This also means that our samples are shaped by the limitations to keywords-based data collection [27, 57, 79].”
Twitter's Glass Ceiling: The Effect of Perceived Gender on Online Visibility (2016, background): “Fourth, the generality of our results, which are from Twitter, across different social media platforms is yet to be seen (Tufekcioglu 2014).”
Characterizing the Demographics Behind the #BlackLivesMatter Movement (2015, result): “In contrast with previous work that reports demographic information by automatically predicting demographic factors for each user based e.g. on their profile picture or name [13, 18, 9, 14], we crowdsourced these annotations.”
Does ‘bigger’ mean ‘better’? Pitfalls and shortcuts associated with big data for social research (2015, background): “Therefore, a further reﬂection should be considered: concentrating mainly, or, even worse, only, on a certain manner of online interaction equates to considering an issue as being conﬁned within a single social network, with no possibility of expanding outsideit(Tufekci2014).”
Analyzing the Digital Traces of Political Manipulation: The 2016 Russian Interference Twitter Campaign (2018, background): “Findings that focused on the positive effects of social media, such as increasing voter turnout [10] or exposure to diverse political views [7] contributed to the general praise of these platforms as a tool for promoting democracy and civic engagement [24], [38], [51], [54], [55].”
Algorithms at Work: The New Contested Terrain of Control (2020, background): “…forms of recording under technical and bureaucratic control, however, since algorithmic recording greatly expands previous con-trol mechanisms in scope and frequency, workers may experience a loss of privacy (Anteby & Chan, 2018; Fourcade & Healy, 2016; Rosenblat & Stark, 2016; Tufekci, 2014).”
Surpassing the Limit: Keyword Clustering to Improve Twitter Sample Coverage (2015, background): “The combination of these favorable traits, found almost uniquely in Twitter, have allowed it to become organically selected as the “’model-organism’ for research with social media data” [21].”
Non-Discriminatory Machine Learning through Convex Fairness Criteria (2018, background): “2016),(Tufekci 2014),(Morstatter and Liu 2017),(Hajian, Bonchi, and Castillo 2016).”
SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours (2017, methodology): “However, Semeval-2015 Task3 is clearly a question answering task, the platform itself supporting a QA format in contrast with the more free-form format of conversations in Twitter.”
Protest Activity Detection and Perceived Violence Estimation from Social Media Images (2017, background): “Work in this field is therefore dominated by formal and qualitative models [23, 27, 29, 45] or surveys of protest participants after a protest occurs [31, 44].”