Auditing Search Engines for Differential Satisfaction Across Demographics

By Rishabh Mehrotra, Ashton Anderson, Fernando Diaz, Amit Sharma, Hanna M. Wallach, Emine Yilmaz. 2017

90 citations recorded by the source corpus.

Many online services, such as search engines, social media platforms, and digital marketplaces, are advertised as being available to any user, regardless of their age, gender, or other demographic factors. However, there are growing concerns that these services may systematically underserve some groups of users. In this paper, we present a framework for internally auditing such services for differences in user satisfaction across demographic groups, using search engines as a case study. We first explain the pitfalls of naively comparing the behavioral metrics that are commonly used to evaluate search engines. We then propose three methods for measuring latent differences in user satisfaction from observed differences in evaluation metrics. To develop these methods, we drew on ideas from the causal inference literature and the multilevel modeling literature. Our framework is broadly applicable to other online services, and provides general insight into interpreting their evaluation metrics.

Subfields: Communication, Information Systems and Management, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Communication

In Information Systems and Management

In Sociology and Political Science

Citations in the corpus

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “Moreover, these surrogate objectives themselves might also be based on imprecise measurement or biased modeling (Mehrotra et al., 2017), and have the potential to create self-fulfilling feedback loops when decisions are made based on the inference results and the outcomes are fed back into the model”
Do News Consumers Want Explanations for Personalized News Rankings (2017, methodology): “characteristics of previously completed tasks such as category, reward or allocated time [8, 17].”
Equity of Attention: Amortizing Individual Fairness in Rankings (2018, background): “A large search engine has been investigated for presence of differential quality of results across demographic groups [27].”
Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems (2018, background): “Recent work has explored the development of classification models with fairness-aware regularization [30], fairness aware decision making systems [9], prediction models for individual fairness [6], fair division of resources [2], auditing for fairness [17] and fairness in rankings [4, 25].”