Characterising Bias in Compressed Models

By Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, Emily L. Denton. 2020

211 citations recorded by the source corpus.

The popularity and widespread use of pruning and quantization is driven by the severe resource constraints of deploying deep neural networks to environments with strict latency, memory and energy requirements. These techniques achieve high levels of compression with negligible impact on top-line metrics (top-1 and top-5 accuracy). However, overall accuracy hides disproportionately high errors on a small subset of examples; we call this subset Compression Identified Exemplars (CIE). We further establish that for CIE examples, compression amplifies existing algorithmic bias. Pruning disproportionately impacts performance on underrepresented features, which often coincides with considerations of fairness. Given that CIE is a relatively small subset but a great contributor of error in the model, we propose its use as a human-in-the-loop auditing tool to surface a tractable subset of the dataset for further inspection or annotation by a domain expert. We provide qualitative and quantitative support that CIE surfaces the most challenging examples in the data distribution for human-in-the-loop auditing.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

On The Impact of Machine Learning Randomness on Group Fairness (2023, background): “[28] Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton.”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “Building upon this result, we ask several followup questions: does the genderedness of a conversation decrease or increase as it proceeds? Do bigger models have more statistical gender bias than smaller ones (Bender et al., 2021; Hooker et al., 2020)? Do our models’ gender biases intersect with raci”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “…different examples might exhibit, including but not limited to: noisy (Wu et al., 2020; Yi and Wu, 2019; Thulasidasan et al., 2019a;b), atypical (Hooker et al., 2020; Buolamwini and Gebru, 2018; Hashimoto et al., 2018; S(cid:32)lowik and Bottou, 2021), challenging (Ahia et al., 2021; Baldock et…”
Intriguing Properties of Compression on Multilingual Models (2022, background): “…numbers (Gale et al., 2019; Li et al., 2020; Hou et al., 2020; Chen et al., 2021; Bai et al., 2020; ab Tessera et al., 2021), the impact on individual sub-populations in the data, such as low-resource languages, can be far more severe (Hooker et al., 2019; Hooker et al., 2020; Ahia et al., 2021).”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, background): “Recent work restricted to the computer vision domain has found that sparse models with comparable top-line performance metrics diverge considerably in behavior on the long-tail of the distribution and are sensitive to distribution shifts (Hooker et al., 2020a; Liebenwein et al., 2021).”
Moving beyond “algorithmic bias is a data problem” (2021, background): “11,12 These are not the only design choices that matter—even more subtle choices like learning rate and length of training can also disproportionately impact error rates on the long-tail of the data-set.”
How Well Do Sparse ImageNet Models Transfer? (2021, background): “Yet, we wish to highlight recent work [29, 30, 44] which examines the robustness of pruned models to input perturbations, as well as the impact of pruning on accuracy of specific segments of the data.”
Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better (2021, background): “Finally, we would also like to emphasize paying attention to the performance of deep learning models (optimized or not) on under-represented classes and out-of-distribution data to ensure model fairness, since quality metrics alone might not be sufficient for discovering deeper issues with models [6”
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts (2022, background): “On the other hand, it has been shown that pruning models tends to cause low-resource groups to be forgotten [46], causing performance to disproportionally drop for some subgroups.”
Randomness In Neural Network Training: Characterizing The Impact of Tooling (2021, background): “Design choices such as compression [Hooker et al., 2020] and privacy [Cummings et al., 2019] can impact disparate impact on sensitive attributes.”
Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers (2021, result): “This ﬁnding is in line with contemporary work by Hooker et al. (2020), who show that distillation in vision models disproportionately harms underrepresented groups.”
Undesirable biases in NLP: Averting a crisis of measurement (2022, background): “, model compression may amplify existing biases (Hooker et al., 2020)).”
Your fairness may vary: Pretrained language model fairness in toxic text classification (2021, background): “On the other hand, for image classiﬁcation models that use large neural networks, Hooker et al. (2020) discuss how model pruning can lead to more biased predictions.”
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021, result): “However, a recent work (Hooker et al., 2020) strongly contradicts this assumption.”
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle (2019, methodology): “Similarly, Hooker et al. [27] show how prioritizing compact models (e.g., with methods such as pruning ) can amplify performance disparities on data with underrepresented attributes.”