Fairness in Recommendation Ranking through Pairwise Comparisons

By Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H. Chi and 1 more. 2019

457 citations recorded by the source corpus.

Recommender systems are one of the most pervasive applications of machine learning in industry, with many services using them to match users to products or information. As such it is important to ask: what are the possible fairness risks, how can we quantify them, and how should we address them? In this paper we offer a set of novel metrics for evaluating algorithmic fairness concerns in recommender systems. In particular we show how measuring fairness based on pairwise comparisons from randomized experiments provides a tractable means to reason about fairness in rankings from recommender systems. Building on this metric, we offer a new regularizer to encourage improving this metric during model training and thus improve fairness in the resulting rankings. We apply this pairwise regularization to a large-scale, production recommender system and show that we are able to significantly improve the system's pairwise fairness.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

Citations in the corpus

Beyond Individual and Group Fairness (2020, background): “There has also been work, however, studying fairness criteria and algorithms for ranking [Celis et al., 2018, Beutel et al., 2019, Narasimhan et al., 2019] and clustering problems [Chierichetti et al., 2017, Schmidt et al., 2018, Backurs et al., 2019] with requirement of a proportional…”
Toward a better trade-off between performance and fairness with kernel-based distribution matching (2019, methodology): “To optimize the fairness metric, [3] adds a MinDiff loss over random pairs of items ( x 1 , x 2 ) with ( y 1 = 1 , y 2 = 0 ) by penalizing the correlation between the following quantities: Our algorithm re-uses this formulation but penalizes the dependence between α and β with MMD .”
Multistakeholder recommendation: Survey and research directions (2020, background): “1 below as well as recent work in fairness in ranking (Zehlike et al. 2017a, b; Singh and Joachims 2018; Beutel et al. 2019).”
User-oriented Fairness in Recommendation (2021, background): “[7] show how to measure fairness based on pairwise comparisons from randomized experiments, and offer a regularizer to improve fairness when training recommendation models.”
Fairness in Machine Learning: A Survey (2020, background): “Subsequent publications expanded this view of recommender fairness by proposing new metrics and algorithms [12, 29, 59, 272, 303, 317].”
A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions (2021, methodology): “Specifically, according to the involved stakeholders, fairness in recommender systems can be divided into two categories [4, 98, 128]: user fairness, which attempts to ensure no algorithmic bias among specific users or demographic groups [9, 90, 96], and item fairness, which indicates fair exposures”
A Review on Fairness in Machine Learning (2022, background): “Beutel et al. [21] propose several measures based on pairwise fairness .”
A Survey on the Fairness of Recommender Systems (2022, methodology): “Grouped by Concept Consistent Fairness CO similar individuals / different groups should receive similar outcomes [30, 36, 39, 48, 53, 54, 91, 93, 95, 97] [6, 28, 45, 68, 73, 86, 90, 107, 112, 116] [12, 22, 25, 43, 44, 52, 64, 85, 104, 114] [8, 20, 58, 60, 61, 75, 76, 78, 83, 94, 102] [82, 99, 106]”