Scalable Learning of Non-Decomposable Objectives

By Elad Eban, Mariano Schain, Alan Mackey, Ariel Gordon, Ryan M. Rifkin, Gal Elidan. 2016

122 citations recorded by the source corpus.

Modern retrieval systems are often driven by an underlying machine learning model. The goal of such systems is to identify and possibly rank the few most relevant items for a given query or context. Thus, such systems are typically evaluated using a ranking-based performance metric such as the area under the precision-recall curve, the $F_\beta$ score, precision at fixed recall, etc. Obviously, it is desirable to train such systems to optimize the metric of interest. In practice, due to the scalability limitations of existing approaches for optimizing such objectives, large-scale retrieval systems are instead trained to maximize classification accuracy, in the hope that performance as measured via the true objective will also be favorable. In this work we present a unified framework that, using straightforward building block bounds, allows for highly scalable optimization of a wide range of ranking-based objectives. We demonstrate the advantage of our approach on several real-life retrieval problems that are significantly larger than those considered in the literature, while achieving substantial improvement in performance over the accuracy-objective baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Wasserstein Fair Classification (2019, methodology): “ing 0:5 ˝ a, where the threshold ˝ ais found using the method of Hardt et al. (2016). This ensures that DP is satisﬁed at threshold ˝= 0:5. Constrained Optimization: Lagrangian-based method (see e.g. Eban et al. (2017); Goh et al. (2016)) using a linear model as the underlying predictor and equal po”
Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goals (2018, methodology): “AUC can be approximated as a set of rate constraints, using the approximation proposed in Eban et al. (2017).”
Two-Player Games for Efficient Non-Convex Constrained Optimization (2018, methodology): “This is indeed a popular approach, having been proposed e.g. for Neyman-Pearson classiﬁ-cation [Davenport et al., 2010, Gasso et al., 2011], more general rate metrics [Goh et al., 2016], and AUC [Eban et al., 2017].”
AUC Maximization in the Era of Big Data and AI: A Survey (2022, methodology): “In particular, Reference [40] proposes to approximate the area under the ROC curve using a Riemann approximation while dividing the range of FPRs into a number of bins where each threshold is associated with a bin.”
Robust Optimization for Fairness with Noisy Protected Groups (2020, background): “[22] Elad Eban, Mariano Schain, Alan Mackey, Ariel Gordon, Rif A.”
Clinically Accurate Chest X-Ray Report Generation (2019, methodology): “Depending on the actual clinical applications, we may, in turn, optimize Recall at Fixed Precision (R@P) or F β score via methods described by Eban et al. (2016).”
The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabeling (2020, methodology): “Weighted Maximum Likelihood was applied to optimize F1 score in [16], and, loss functions have been developed to allow a machine learning model to directly optimize F1 score [17].”