Fairness Without Demographics in Repeated Loss Minimization

By Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, Percy Liang. 2018

661 citations recorded by the source corpus.

Machine learning models (e.g., speech recognizers) are usually trained to minimize average loss, which results in representation disparity---minority groups (e.g., non-native speakers) contribute less to the training objective and thus tend to suffer higher loss. Worse, as model accuracy affects user retention, a minority group can shrink over time. In this paper, we first show that the status quo of empirical risk minimization (ERM) amplifies representation disparity over time, which can even make initially fair models unfair. To mitigate this, we develop an approach based on distributionally robust optimization (DRO), which minimizes the worst case risk over all distributions close to the empirical distribution. We prove that this approach controls the risk of the minority group at each time step, in the spirit of Rawlsian distributive justice, while remaining oblivious to the identity of the groups. We demonstrate that DRO prevents disparity amplification on examples where ERM fails, and show improvements in minority group user satisfaction in a real-world text autocomplete task.

Subfields: Demography, General Health Professions, Statistics and Probability, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Demography

In General Health Professions

In Statistics and Probability

In Safety, Robustness & Interpretability

Citations in the corpus

Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021, background): “Language applications are capable of automating erasure, in a cyclical fashion (Hashimoto et al., 2018; Sap, 2021).”
Societal Biases in Language Generation: Progress and Challenges (2021, background): “…if contributions and subsequent improvements are focused on high-resource languages, this further increases the accuracy gap between the high and low resource languages, diminishing opportunities for speakers of the low resource languages, i.e., representation disparity (Hashimoto et al., 2018).”
Shortcut learning in deep neural networks (2020, background): “In the dynamical setting a related problem is called disparity amplification [74], where sequential feedback loops may amplify a model’s reliance on a majority group.”
Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, background): “data collected, thus worsening the disparity in the data sets (Hashimoto et al., 2018).”
Beyond Individual and Group Fairness (2020, background): “Recent works have also studied the long-term impact of optimizing fairness criteria in settings with feedback mechanisms [Liu et al., 2018, Hashimoto et al., 2018, Mouzannar et al., 2019, Kannan et al., 2019].”
Fair Resource Allocation in Federated Learning (2019, background): “, 2016), another set of works optimize an objective under some explicit fairness constraints during training time (Agarwal et al., 2018; Cotter et al., 2018; Hashimoto et al., 2018; Woodworth et al., 2017; Zafar et al., 2017; 2015).”
Proportionally Fair Clustering (2019, background): “Motivated by concerns ranging from the granting of bail in the legal system to the quality of recommender systems, researchers have devoted considerable effort to developing fair algorithms for the canonical supervised learning tasks of classification and regression (Dwork et al., 2012; Kleinberg et”
Multiaccuracy: Black-Box Post-Processing for Fairness in Classification (2018, background): “Many different notions of fairness have been proposed in literature on learning and classification (Dwork et al. 2012; Hardt, Price, and Srebro 2016; Zemel et al. 2013; Dwork et al. 2017; Hébert-Johnson et al. 2018; Kearns et al. 2017; Hashimoto et al. 2018; Kim, Reingold, and Rothblum 2018; Rothblu”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, positive): “…worse performance on African-American Vernacular English compared to Standard American English on part-of-speech tagging (Jørgensen et al., 2015), dependency parsing (Blodgett et al., 2016), language identification (Blodgett and O’Connor, 2017), and auto-correct systems (Hashimoto et al., 2018).”
Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities (2021, methodology): “Distributionally robust optimisation represents an alternative method for preventing disparity ampliﬁcation, bounding the worst-case risk over groups with unknown group membership by optimising the worst-case risk over an appropriate risk region (Hashimoto et al. 2018).”
Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization (2019, methodology): “However, even models that perform well on average can consistently fail on rare and atypical examples (Hovy & Sgaard, 2015; Blodgett et al., 2016; Tatman, 2017; Hashimoto et al., 2018; Duchi et al., 2019).”
Distributionally Robust Neural Networks (2020, methodology): “, 2015; Duchi & Namkoong, 2016) and to learn models that are robust over different data subsets (Duchi & Namkoong, 2018; Hashimoto et al., 2018).”
Fair Generative Modeling via Weak Supervision (2019, background): “, 2019; Jo & Gebru, 2020) or algorithmic decisions such as the loss function or evaluation metric (Hardt et al., 2016; Buolamwini & Gebru, 2018; Kim et al., 2018; Liu et al., 2018; Hashimoto et al., 2018), among others, may have undesirable consequences.”
Selective Classification Can Magnify Disparities Across Groups (2020, background): “This approach is not a silver bullet: it relies on knowing group identities during training, which are not always available (Hashimoto et al., 2018).”
Debiased Contrastive Learning (2020, background): “For instance, the majority groups are sampled more frequently than the minority ones [16].”
Data Feedback Loops: Model-driven Amplification of Dataset Biases (2022, background): “In this context, existing work has shown that optimizing strictly for ranking metrics such as accuracy can create echo chambers, where minority populations are crowded out and disengage from the platform (Hashimoto et al., 2018; Jiang et al., 2019).”
Gradient Matching for Domain Generalization (2021, positive): “Using vanilla ERM is theoretically unfounded (Hashimoto et al., 2018; Blodgett et al., 2016; Tatman, 2017) since ERM is guaranteed to work only when train and test distributions match.”
DORO: Distributional and Outlier Robust Optimization (2021, positive): “Several recent papers (Hashimoto et al., 2018; Oren et al., 2019; Xu et al., 2020) proposed using DRO to deal with subpopulation shift, but it was also observed that DRO was prone to overﬁt in practice (Sagawa et al., 2020a;b).”