Tilted Empirical Risk Minimization

By Tian Li, Ahmad Beirami, Maziar Sanjabi, Virginia Smith. 2020

155 citations recorded by the source corpus.

Empirical risk minimization (ERM) is typically designed to perform well on the average loss, which can result in estimators that are sensitive to outliers, generalize poorly, or treat subgroups unfairly. While many methods aim to address these problems individually, in this work, we explore them through a unified framework---tilted empirical risk minimization (TERM). In particular, we show that it is possible to flexibly tune the impact of individual losses through a straightforward extension to ERM using a hyperparameter called the tilt. We provide several interpretations of the resulting framework: We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; has variance-reduction properties that can benefit generalization; and can be viewed as a smooth approximation to a superquantile method. We develop batch and stochastic first-order optimization methods for solving TERM, and show that the problem can be efficiently solved relative to common alternatives. Finally, we demonstrate that TERM can be used for a multitude of applications, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance. TERM is not only competitive with existing solutions tailored to these individual problems, but can also enable entirely new applications, such as simultaneously addressing outliers and promoting fairness.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pascale Fung
2. Tian Li
3. Andrea Madotto
4. Virginia Smith
5. Wenliang Dai
6. Ahmad Beirami
7. Shengyuan Hu
8. Nayeon Lee

In Management Science and Operations Research

1. Tian Li
2. Dan Su
3. Virginia Smith
4. Ahmad Beirami
5. Andrea Madotto
6. Nayeon Lee
7. Pascale Fung
8. Jakub Konecný

In Statistics and Probability

1. Dan Su
2. Virginia Smith
3. Tian Li
4. Ahmad Beirami

In Optimization & Training

1. Tian Li
2. Virginia Smith
3. Shengyuan Hu
4. Ahmad Beirami
5. Dan Su
6. Andrea Madotto
7. Etsuko Ishii
8. Nayeon Lee

Citations in the corpus

• Survey of Hallucination in Natural Language Generation (2022, background): “Other general training methods such as regularization [82, 95, 132] and loss reconstruction [107, 193, 199] have also been proposed to tackle the hallucination problem.”
• Ditto: Fair and Robust Federated Learning Through Personalization (2020, methodology): “Fairness of Ditto To explore the fairness of Ditto, we compare against TERM (Li et al., 2021) as a baseline.”
• A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Li et al. [333] explored the problem through an integrated framework called TERM (Tilted Empirical Risk Minimization) to increase or decrease the impact of outliers, respectively.”
• S-SPPO: Semantic-Calibrated Self-Play Preference Optimization (2026, unknown): “Connection to Tilted-ERM.”