Is the Sample Good Enough? Comparing Data from Twitter's Streaming API with Twitter's Firehose

By Fred Morstatter, Jürgen Pfeffer, Huan Liu, Kathleen M. Carley. 2013

1,032 citations recorded by the source corpus.

Twitter is a social media giant famous for the exchange of short, 140-character messages called "tweets". In the scientific community, the microblogging site is known for openness in sharing its data. It provides a glance into its millions of users and billions of tweets through a "Streaming API" which provides a sample of all tweets matching some parameters preset by the API user. The API service has been used by many researchers, companies, and governmental institutions that want to extract knowledge in accordance with a diverse array of questions pertaining to social media. The essential drawback of the Twitter API is the lack of documentation concerning what and how much data users get. This leads researchers to question whether the sampled data is a valid representation of the overall activity on Twitter. In this work we embark on answering this question by comparing data collected using Twitter's sampled API service with data collected using the full, albeit costly, Firehose stream that includes every single published tweet. We compare both datasets using common statistical metrics as well as metrics that allow us to compare topics, networks, and locations of tweets. The results of our work will help researchers and practitioners understand the implications of using the Streaming API.

Subfields: Information Systems, Statistical and Nonlinear Physics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Statistical and Nonlinear Physics

In Evaluation & Benchmarks

Citations in the corpus

A Survey on Bias and Fairness in Machine Learning (2019, background): “The differences and biases in the networks can be a result of many factors, such as network sampling, as shown in [51, 91], which can change the network measures and cause different types of problems.”
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “Yet, APIs may conceal specific details about how the returned data items are sampled from those satisfying a query [boyd and Crawford 2012; Bruns 2013; Maddock et al. 2015; Morstatter et al. 2013].”
Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English (2017, background): “e data is a sample of millions of publicly posted geolocated Twier messages (from the Decahose/Gardenhose stream [17]), most of which are sent on mobile phones, by authors in the U.”
Hate Lingo: A Target-based Linguistic Analysis of Hate Speech in Social Media (2018, background): “Other challenges include overcoming sample quality issues and other issues associated with Twitter Streaming API as discussed by (Tufekci 2014; Morstatter et al. 2013), and the need to move beyond keyword-based methods that have been shown to miss many instances of hateful speech (Saleem et al.”
Assessing the bias in samples of large online networks (2014, background): “…related to commercial products (Jansen et al., 2009), and thers analyse messages labelled with specific hashtags (Borgeolthoefer et al., 2011; Morstatter et al., 2013; Romero et al., 2011). hese studies reveal that the dynamics of communication change cross different information domains;…”
Predictive Analysis on Twitter: Techniques and Applications (2018, background): “.com/special-columns/adhutchinson/2015-09-09/big-brandtheory-loreal-stays-connected-their-audience Predictive Analysis on Twitter: Techniques and Applications 27 by incorporating background knowledge [85,20]. Specically, monitoring the engagement and public opinion about ongoing events from temporal”
Geo-located Twitter as proxy for global mobility patterns (2013, methodology): “The stream was gathered through the Twitter Streaming API.2 Although the service sets a limit on how much data can be accessed to less than 1% of the total Twitter stream, the total geo-located content was found not to exceed this restriction (Morstatter et al. 2013).”
When is it biased?: assessing the representativeness of twitter's streaming API (2014, methodology): “This is the approach taken in [15].”
Toward Reproducibility in Online Social Network Research (2018, background): “The bit attribute immediately sanitises the response from the Facebook API to only return 1 if the attribute is present, or 0 if it is not, before the data are made available to the experimental application.”
Twitter as Data (2018, background): “Comparisons of the stream to the complete Twitterverse find the stream does not differ in a meaningful manner (Morstatter et al., 2013; Valkanas et al., 2014).”
Surpassing the Limit: Keyword Clustering to Improve Twitter Sample Coverage (2015, background): “They explain that according to the “law of large numbers” if the sample is truely random then it should relate well with the results from the population, in this case the “firehose” results [19].”
Cybercrime detection in online communications: The experimental case of cyberbullying detection in the Twitter network (2016, methodology): “Morstatter et al. (2013) analyzed whether the data extracted from this API could sufficiently represent the activities in the Twitter network in general, and determined that when geo-code filtering was used, API would return a nearly complete set of geo-tagged tweets despite the geo-coded sampling…”
Online Human-Bot Interactions: Detection, Estimation, and Characterization (2017, methodology): “For each of these accounts, we collected their tweets through the Twitter Search API.”
The Tweets They Are a-Changin: Evolution of Twitter Users and Behavior (2014, methodology): “To measure the limitations and representation of sample data, recent work (Morstatter et al. 2013) examined how Twitter selected tweets to re-”
Can One Tamper with the Sample API?: Toward Neutralizing Bias from Spam and Bot Content (2016, result): “By comparing the results of a Firehose crawl with a Filter API crawl, the authors of one study [4] discover that bias in the way tweets are sampled yields completely diﬀerent statistics, e.g. the top hashtags in the data and the topics in the text.”