Satisfying Real-world Goals with Dataset Constraints

By Gabriel Goh, Andrew Cotter, Maya R. Gupta, Michael P. Friedlander. 2016

218 citations recorded by the source corpus.

The goal of minimizing misclassification error on a training set is often just one of several real-world goals that might be defined on different datasets. For example, one may require a classifier to also make positive predictions at some specified rate for some subpopulation (fairness), or to achieve a specified empirical recall. Other real-world goals include reducing churn with respect to a previously deployed model, or stabilizing online training. In this paper we propose handling multiple goals on multiple datasets by training with dataset constraints, using the ramp penalty to accurately quantify costs, and present an efficient algorithm to approximately optimize the resulting non-convex constrained optimization problem. Experiments on both benchmark and real-world industry datasets demonstrate the effectiveness of our approach.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Wasserstein Fair Classification (2019, methodology): “We preprocessed the data in the same way as done in Goh et al. (2016); Zafar et al.”
Stable and Fair Classification (2019, background): “Another approach for fair classiﬁcation is to shift the decision boundary of a baseline classiﬁer, e.g., [28, 37, 34, 61, 73, 22].”
50 Years of Test (Un)fairness: Lessons for Machine Learning (2018, background): “It is closely related to coverage constraints [29], class mass normalization [70] and expectation regularization [45].”
On Fairness and Calibration (2017, background): “This notion of non-discrimination is feasible in many settings, and researchers have developed tractable algorithms for achieving it [17, 19, 34, 37].”
Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment (2016, neutral): “ion, making it a more appealing notion of fairness. Related Work. There have been a number of studies, including our own prior work [28], proposing methods for detecting [10, 21, 23, 25] and removing [9, 10, 13, 16, 17, 23, 28, 29] unfairness when it is dened in terms of disparate treatment, dispara”
The Promise and Peril of Human Evaluation for Model Interpretability (2017, methodology): “[8] proposes a solution for this problem using dataset constraints and a ramp penalty.”
Learning Optimized Risk Scores (2016, background): “Current approaches often aim to address a specific kind of constraints for generic models via pre- or post-processing (see e.g., Goh et al., 2016; Calmon et al., 2017).”
From Parity to Preference-based Notions of Fairness in Classification (2017, neutral): “parity in impact [7, 18, 21] or both [12, 14, 17, 20, 32, 33, 34].”
Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goals (2018, methodology): “x with 0), requires enumerating over the entire constraint dataset, making this incompatible with the use with a computationally-cheap stochastic gradient optimizer. 3. How To Use Rate Constraints In Goh et al. (2016), we showed that a number of useful machine learning goals can be expressed as cons”
Algorithmic Fairness (2020, background): “These mechanisms involve modifying the machine learning algorithms to account for fairness during the training time [4, 12, 13, 28, 59, 79, 138, 142, 143].”
A Review on Fairness in Machine Learning (2022, methodology): “[84] In-process SVM Disparate impact, Equal opportunity Disparate impact Dataset Constraints Adult Constrained non-convex optimization using SVM with ramp loss, to enhance disparate impact.”
Unleashing Linear Optimizers for Group-Fair Learning and Optimization (2018, background): “Despite such results, we know that we can design multi-objective metrics that trade-off fairness for accuracy and vice-versa (Woodworth et al. (2017); Menon and Williamson (2017); Dwork et al. (2018); Goh et al. (2016)).”
A General Approach to Fairness with Optimal Transport (2020, background): “This has motivated researchers to investigate techniques for ensuring that learned models satisfy fairness properties (Dwork et al. 2012; Feldman et al. 2015; Goh et al. 2016; Chouldechova 2017; Corbett-Davies et al. 2017; Gajane and Pechenizkiy 2017; Kusner et al. 2017; Cotter et al. 2018; Mitchell”
Training Well-Generalizing Classifiers for Fairness Metrics and Other Data-Dependent Constraints (2018, background): “Appendix B† contains two examples of how Equation 1 can be used to express certain data-dependent constraints (see Goh et al. (2016); Narasimhan (2018) for more).”
Toward a better trade-off between performance and fairness with kernel-based distribution matching (2019, background): “Numerous regularization approaches have been proposed to encourage models during training to optimize for various fairness deﬁnitions [20, 4, 18, 1, 21, 11].”
Nonconvex Optimization for Regression with Fairness Constraints (2018, background): “…papers in the latter category (Kamiran & Calders, 2010; Zliobaite et al., 2011; Kamishima et al., 2012a; Ristanoski et al., 2013; Fish et al., 2015; Hardt et al., 2016; Goh et al., 2016; Zafar et al., 2017a) deal with classification tasks, and thus cannot directly deal with regression tasks.”