Computational social scientist beware: Simpson’s paradox in behavioral data

By Kristina Lerman. 2017

32 citations recorded by the source corpus.

Observational data about human behavior are often heterogeneous, i.e., generated by subgroups within the population under study that vary in size and behavior. Heterogeneity predisposes analysis to Simpson’s paradox, whereby the trends observed in data that have been aggregated over the entire population may be substantially different from those of the underlying subgroups. I illustrate Simpson’s paradox with several examples coming from studies of online behavior and show that aggregate response leads to wrong conclusions about the underlying individual behavior. I then present a simple method to test whether Simpson’s paradox is affecting results of analysis. The presence of Simpson’s paradox in social data suggests that important behavioral differences exist within the population, and failure to take these differences into account can distort the studies’ findings.

Subfields: Communication, Statistical and Nonlinear Physics, Statistics and Probability, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Communication

In Statistical and Nonlinear Physics

In Statistics and Probability

In Social Sciences & Humanities

Citations in the corpus

A Survey on Bias and Fairness in Machine Learning (2019, background): “Simpson’s paradox has been observed in a variety of domains, including biology [34], psychology [68], astronomy [89], and computational social science [77].”
Using Simpson's Paradox to Discover Interesting Patterns in Behavioral Data (2018, methodology): “Digital traces of activity have exposed human behavior to quantitative analysis (Lazer et al. 2009; McFarland, Lewis, and Goldberg 2016).”