What is the Effect of Importance Weighting in Deep Learning?

By Jonathon Byrd, Zachary Chase Lipton. 2018

535 citations recorded by the source corpus.

Importance-weighted risk minimization is a key ingredient in many machine learning algorithms for causal inference, domain adaptation, class imbalance, and off-policy reinforcement learning. While the effect of importance weighting is well-characterized for low-capacity misspecified models, little is known about how it impacts over-parameterized, deep neural networks. This work is inspired by recent theoretical results showing that on (linearly) separable data, deep linear networks optimized by SGD learn weight-agnostic solutions, prompting us to ask, for realistic deep networks, for which many practical datasets are separable, what is the effect of importance weighting? We present the surprising finding that while importance weighting impacts models early in training, its effect diminishes over successive epochs. Moreover, while L2 regularization and batch normalization (but not dropout), restore some of the impact of importance weighting, they express the effect via (seemingly) the wrong abstraction: why should practitioners tweak the L2 regularization, and by how much, to produce the correct weighting effect? Our experiments confirm these findings across a range of architectures and datasets.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “(7) Both of these baselines work in the setting where the training loss is non-vanishing, which typically requires strong regularization or early stopping when using overparameterized neural networks (Byrd and Lipton, 2019; Sagawa et al., 2020,).”
Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting (2019, background): “Density ratios enjoy widespread use across machine learning e.g., for handling covariate shifts, class imbalance etc. [9, 34].”
Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization (2019, background): “Here, we show theoretically and empirically that DRO also outperforms a strong importance weighting baseline that is commonly used in machine learning tasks where the train and test distributions differ (Shimodaira, 2000; Byrd & Lipton, 2019).”
Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets (2020, background): “To achieve a more balanced distribution, researchers have proposed to either over-sample the minority classes [30,2,1], or under-sample the majority classes [17,10,1].”
Undoing the Damage of Label Shift for Cross-domain Semantic Segmentation (2022, background): “As stated in [4, 33], while re-weigting the loss with the label distribution ratio also helps to guide the classifier to approach the target classifier, yet not as effectively as directly using it to correct the classifier output like our classifier refinement loss.”
Improving Calibration for Long-Tailed Recognition (2020, background): “There are two groups of re-sampling strategies: over-sampling the tail-class images [28, 2, 3] and under-sampling the head-class images [13, 2].”
Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss (2019, background): “In a recent work [6], Byrd and Lipton study the effect of importance weighting and show that empirically importance weighting does not have a significant effect when no regularization is applied, which is consistent with the theoretical prediction in [48] that logistical regression without regulariz”
Just Train Twice: Improving Group Robustness without Training Group Information (2021, methodology): “…2020) 44 The sensitivity of CVaR DRO, LfF, and J TT to hyperparam-eters is consistent with prior work showing that reweighting methods like these require appropriate capacity control via techniques like early stopping or strong (cid:96) 2 regularization (Byrd & Lipton, 2019; Sagawa et al., 2020a).”
CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning (2021, methodology): “Various solutions have been proposed to help alleviate bias, such as re-sampling [3, 4], re-”
Fair Generative Modeling via Weak Supervision (2019, methodology): “Reweighting datapoints is a common algorithmic technique for problems such as dataset bias and class imbalance [41].”
Rethinking the Value of Labels for Improving Class-Imbalanced Learning (2020, background): “In addition, cost-sensitive re-weighting schemes [6, 22, 23, 27, 52] are proposed to (dynamically) adjust weights during training for different classes or even different samples.”
IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies (2023, unknown): “However, it is known in the literature that using highly expressive models with importance weighted objectives can be problematic, as such models can increase the likelihood of all training points regardless of their weight [6, 48].”
The Devil is in Classification: A Simple Framework for Long-tail Instance Segmentation (2020, background): “Two major approaches tackling long-tail problems are sampling [4],[5], [16],[36] and loss re-weighting [10],[18],[19].”
Parametric Contrastive Learning (2021, background): “The most classical way to deal with long-tailed datasets is to over-sample lowfrequency class images [41, 56, 2, 3] or under-sample highfrequency class images [20, 27, 2].”
Balanced Meta-Softmax for Long-Tailed Visual Recognition (2020, methodology): “Recent works [29, 3] show that the global minimum of the Softmax regression is independent of the mini-batch sampling process.”
Forest R-CNN: Large-Vocabulary Long-Tailed Object Detection and Instance Segmentation (2020, methodology): “One prominent method for long-tailed visual recognition is data re-balancing, which is usually divided into two groups: resampling [2, 15, 18, 33, 46] and re-weighting [8, 9, 19].”
Remix: Rebalanced Mixup (2020, background): “Furthermore, it has been shown that re-weighting is not effective when no regularization is applied [4].”
An Investigation of Why Overparameterization Exacerbates Spurious Correlations (2020, methodology): “…error, prior work proposed modified objectives that focus on the worst-group loss, such as group distributionally robust optimization (group DRO) which directly optimizes for the worst-group training loss (Hu et al., 2018; Sagawa et al., 2020) or reweighting (Shimodaira, 2000; Byrd & Lipton, 2019).”
Balanced Activation for Long-tailed Visual Recognition (2020, unknown): “Most state-of-the-art (SOTA) methods use the class-balanced sampling or loss re-weighting to “simulate" a balanced training set [1,10].”