Effects of Sampling on Twitter Trend Detection
By Andrew Yates, A. Kolcz, Nazli Goharian, Ophir Frieder. 2016
7 citations recorded by the source corpus.
Much research has focused on detecting trends on Twitter, including health-related trends such as mentions of Influenza-like illnesses or their symptoms. The majority of this research has been conducted using Twitter’s public feed, which includes only about 1% of all public tweets. It is unclear if, when, and how using Twitter’s 1% feed has affected the evaluation of trend detection methods. In this work we use a larger feed to investigate the effects of sampling on Twitter trend detection. We focus on using health-related trends to estimate the prevalence of Influenza-like illnesses based on tweets. We use ground truth obtained from the CDC and Google Flu Trends to explore how the prevalence estimates degrade when moving from a 100% to a 1% sample. We find that using the 1% sample is unlikely to substantially harm ILI estimates made at the national level, but can cause poor performance when estimates are made at the city level.
Subfields: General Social Sciences
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In General Social Sciences
Citations in the corpus
- Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, methodology): “Data from these APIs have been compared against the full data stream, finding statistical disparities (Yates et al., 2016).”
- Tampering with Twitter’s Sample API (2018, methodology): “Several studies discuss working, compositions and possible biases of data [47, 48] and a “reverse-engineered” model has been developed for the Sample API, which indicates that the sampling is based on a millisecond time window and that the timestamp at which the Tweet arrived at Twitter’s servers is”