Can you Trust the Trend?: Discovering Simpson's Paradoxes in Social Data

By Nazanin Alipourfard, Peter G. Fennell, Kristina Lerman. 2018

36 citations recorded by the source corpus.

We investigate how Simpson»s paradox affects analysis of trends in social data. According to the paradox, the trends observed in data that has been aggregated over an entire population may be different from, and even opposite to, those of the underlying subgroups. Failure to take this effect into account can lead analysis to wrong conclusions. We present a statistical method to automatically identify Simpson»s paradox in data by comparing statistical trends in the aggregate data to those in the disaggregated subgroups. We apply the approach to data from Stack Exchange, a popular question-answering platform, to analyze factors affecting answerer performance, specifically, the likelihood that an answer written by a user will be accepted by the asker as the best answer to his or her question. Our analysis confirms a known Simpson»s paradox and identifies several new instances. These paradoxes provide novel insights into user behavior on Stack Exchange.

Subfields: Artificial Intelligence, Communication, Information Systems, Statistics and Probability, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Information Systems

In Statistics and Probability

In Social Sciences & Humanities

Citations in the corpus

A Survey on Bias and Fairness in Machine Learning (2019, methodology): “o identify instances of this unfairness in a particular dataset. Inroads toward this problem have been made in the areas of data bias by detecting instances of Simpson’s Paradox in arbitrary datasets [2]; however, unfairness may require more consideration due to the variety of definitions and the nu”
Using Simpson's Paradox to Discover Interesting Patterns in Behavioral Data (2018, methodology): “Simply partitioning the data into fixed-size bins (Alipourfard, Fennell, and Lerman 2018), or percentiles, can be problematic when Xc has a heavy-tailed distribution, since the bins covering the tail will have few data points in them.”
Computational social scientist beware: Simpson’s paradox in behavioral data (2017, background): “As a result, the data may appear to exhibit a trend, which disappears or reverses when the data are disaggregated by subgroups [1].”