Intriguing Properties of Compression on Multilingual Models

By Kelechi Ogueji, Orevaoghene Ahia, Gbemileke Onilude, Sebastian Gehrmann, Sara Hooker, Julia Kreutzer. 2022

16 citations recorded by the source corpus.

Multilingual models are often particularly dependent on scaling to generalize to a growing number of languages. Compression techniques are widely relied upon to reconcile the growth in model size with real world resource constraints, but compression can have a disparate effect on model performance for low-resource languages. It is thus crucial to understand the trade-offs between scale, multilingualism, and compression. In this work, we propose an experimental framework to characterize the impact of sparsifying multilingual pre-trained language models during fine-tuning.Applying this framework to mBERT named entity recognition models across 40 languages, we find that compression confers several intriguing and previously unknown generalization properties. In contrast to prior findings, we find that compression may improve model robustness over dense models. We additionally observe that under certain sparsification regimes compression may aid, rather than disproportionately impact the performance of low-resource languages.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “These can include choices about many stages of the optimization process [237, 129]; privacy constraints [24], widely used compression techniques [109, 15, 169] and the choice hardware [273] have all been found to amplify harm on underrepresented protected attributes [28].”
FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensembling (2023, background): “…focus on datasets collection and curation (Barocas et al., 2019; Zhao et al., 2017; Shankar et al., 2017), with limited work to-date understanding the role of model design or optimization choices on amplifying or curbing bias (Ogueji et al., 2022; Hooker et al., 2019; Balestriero et al., 2022).”