The Parable of Google Flu: Traps in Big Data Analysis

By David M. J. Lazer, Ryan Kennedy, Gary King, Alessandro Vespignani. 2014

2,270 citations recorded by the source corpus.

Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data. In February 2013, Google Flu Trends (GFT) made headlines but not for a reason that Google executives or the creators of the flu tracking system would have hoped. Nature reported that GFT was predicting more than double the proportion of doctor visits for influenza-like illness (ILI) than the Centers for Disease Control and Prevention (CDC), which bases its estimates on surveillance reports from laboratories across the United States (1, 2). This happened despite the fact that GFT was built to predict CDC reports. Given that GFT is often held up as an exemplary use of big data (3, 4), what lessons can we draw from this error?

Subfields: Artificial Intelligence, Epidemiology, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Epidemiology

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2016, methodology): “two, as measured by the number of influenza-related doctor visits in subsequent years, according to CDC data [15].”
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “It also includes temporal validity , which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
The ethics of algorithms: Mapping the debate (2016, neutral): “The search for causal links is diﬃcult, as correlations established in large, proprietary datasets are frequently not reproducible or falsiﬁable (cf. Ioannidis, 2005; Lazer et al., 2014).”
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI (2020, neutral): “the original classification task [42, 46].”
Can we trust Big Data? Applying philosophy of science to software (2016, methodology): “Because of this, the project is often taken to exemplify ‘‘Big Data hubris’’ (Lazer et al., 2014), the often underlying assumption that large amounts of data and the patterns that are discovered through its analysis can yield results independently from or without the aid of principled theoretical…”
Opportunities in Machine Learning for Healthcare (2018, positive): “n for the new population(Tsymbal, 2004). Internal Validity - Shift over time. In a notable example of concept drift, Google Flu Trends persistently overestimated ﬂu due to shifts in search behaviors (Lazer et al., 2014). In this case, the initial model was a great success, leveraging Google search d”
Text as Data (2017, methodology): “Lazer et al. (2014), for example, show that the accuracy of the Google Flu Trends model—which has not been recalibrated or updated based on more recent data—has deteriorated dramatically, and that in recent years it is outperformed by simple extrapolation from prior CDC estimates.”
A Hierarchy of Limitations in Machine Learning (2020, background): “Lazer et al. (2014) phrase this as a problem of overfitting, and indeed we can think of models that fit to anything other than an underlying causal process as “overfitting,” but it is more strongly a lesson about causality (Harford, 2014).”
Leveraging Facebook's Advertising Platform to Monitor Stocks of Migrants (2017, background): “These approaches also stimulated reflection about the problem of relying solely on non-representative sources, when in fact the combination of traditional and new data, within a unified framework, would yield more accurate estimates and predictions (Lazer et al. 2014).”
Heading toward Artificial Intelligence 2.0 (2016, background): “[20] suggested that the fusion and intelligent analysis of multiple sources of information were the characteristics associated with knowledge and system evolution and were the necessary means of solving “big data hubris.”
Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case (2020, methodology): “While initially regarded as a great success, GFT suffered from over-estimation of peak ILI magnitude in subsequent years (Olson et al., 2013; Lazer et al., 2014).”
The Role of Big Data and Predictive Analytics in Retailing (2017, background): “Later a deeper analysis showed that these 'theory-free' projections were overpredicting actual cases by 100%, among other problems that arose (see Lazer et al. 2014 for a review).”